Why would certain packets not reach nat PREROUTING chain?

Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 1294 bytes --]

I'm troubeshooting an issue of accessing a VPN through NAT. Right now the
problem can be reduced to the following question:

Under what conditions would inbound packets not be routing through the nat
PREROUTING chain?

These packets are arriving on inbound UDP port 500. They show up with
tcpdump, but when I add a log rule, e.g.

iptables -t nat -I PREROUTING -p udp -j LOG

The packets are not logged.  (They are also not DNAT'd to the proper
internal host, but that makes sense if they're not reaching the PREROUTING
chain at all).

I have nothing in the mangle table.

I am not running any IPSec services on the NAT box.

There is nothing between the NAT box and the Internet.

Most of the iptables tutorials warn against filtering in nat PREROUTING,
because "it will be bypassed in certain cases."  But what cases are those?

The iptables LOG targets are working generally--traffic coming from the
internal client to the NAT box and then the NAT box to the external VPN
server are all logged.  The only thing that is not being logged--and
presumably not arriving at the nat PREROUTING chain--are the inbound
packets.  Yet they are definitely arriving, as tcpdump -i eth0 indicates.

Any suggestions?
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org




[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 494 bytes --]

Adam Rosi-Kessel wrote:
> I'm troubleshooting an issue of accessing a VPN through NAT. Right now the
> problem can be reduced to the following question:
> Under what conditions would inbound packets not be routing through the nat
> PREROUTING chain?

I should add that, just for debugging purposes, the default policy for all
chains is set to ACCEPT. There are also no DROP rules anywhere in any table
(again, just for debugging).
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Alexander Samad]

[-- Attachment #1: Type: text/plain, Size: 732 bytes --]

On Wed, Nov 09, 2005 at 08:59:37PM -0500, Adam Rosi-Kessel wrote:
> Adam Rosi-Kessel wrote:
> > I'm troubleshooting an issue of accessing a VPN through NAT. Right now the
> > problem can be reduced to the following question:
> > Under what conditions would inbound packets not be routing through the nat
> > PREROUTING chain?
> 
> I should add that, just for debugging purposes, the default policy for all
> chains is set to ACCEPT. There are also no DROP rules anywhere in any table
> (again, just for debugging).

my understanding is that the NAT table only sees the initial packets of
a connection

if this is ipsec it could be a ipsec problem ?

> -- 
> Adam Rosi-Kessel
> http://adam.rosi-kessel.org
> 



[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

On Thu, Nov 10, 2005 at 02:18:34PM +1100, Alexander Samad wrote:
> On Wed, Nov 09, 2005 at 08:59:37PM -0500, Adam Rosi-Kessel wrote:
> > Adam Rosi-Kessel wrote:
> > > I'm troubleshooting an issue of accessing a VPN through NAT. Right now the
> > > problem can be reduced to the following question:
> > > Under what conditions would inbound packets not be routing through the nat
> > > PREROUTING chain?
> > I should add that, just for debugging purposes, the default policy for all
> > chains is set to ACCEPT. There are also no DROP rules anywhere in any table
> > (again, just for debugging).
> my understanding is that the NAT table only sees the initial packets of
> a connection

Is that the only situation in which an inbound packet that shows up in
tcpdump would not show up in an itables nat PREROUTING log?

I also tried adding a raw table entry to prevent connection tracking, as
follows:

iptables -t raw -A PREROUTING -p udp --dport 500 -j NOTRACK

But the iptables nat PREROUTING log still did not show any of the inbound
packets.

> if this is ipsec it could be a ipsec problem ?

Well, the NAT box is not running IPSec. I'm trying to diagnose an IPSec
problem involving the client and the server, but for various reasons I
can't make the NAT box the IPSec endpoint. It shouldn't be that difficult
because I only have one box inside the LAN with a fixed NAT IP address
that needs to connect to the VPN server, and so I'm trying to direct udp
500 right to that client, but the problem seems to be the inbound packets
are not even entering the PREROUTING chain.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 214 bytes --]

George Alexandru Dragoi wrote:
> Better use mangle table, all packets traverse it. And doesn't require
> conntrack.

But I can't do DNAT in mangle, right?
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Sandro Dentella]

On Wed, Nov 09, 2005 at 08:57:45PM -0500, Adam Rosi-Kessel wrote:
> I'm troubeshooting an issue of accessing a VPN through NAT. Right now the
> problem can be reduced to the following question:
> 
> Under what conditions would inbound packets not be routing through the nat
> PREROUTING chain?

That's a problem that puzzles me too. Do you have fancy routing tables?
(several different tables setup w/ iproute2).

I also have a setup in which icmp packets will not get to PREROUTING. My
understanding is that the kernel does not understand they are destined for
that box: could that be your situation?

In my stup, ifconfig eth3:
eth3      Link encap:Ethernet  HWaddr 00:0A:5E:59:EF:09
          inet addr:192.168.111.1  Bcast:192.168.111.255  Mask:255.255.255.0

but these packets 

10:18:07.676131 IP 192.168.111.1 > 217.27.90.70: icmp 64: echo request seq 213
10:18:07.726977 IP 217.27.90.70 > 192.168.111.1: icmp 64: echo reply seq 213

do not enter PREROUTING and are just discarded 

In my case this is due to a peculiar routing tables setup,
http://mailman.ds9a.nl/pipermail/lartc/2005q4/017168.html that I have not
been able to debug: I didn't receice any hint on this list, lartc and
netdev. Is it such an obscure matter?


sandro



-- 
Sandro Dentella  *:-)
e-mail: sandro@e-den.it 
http://www.tksql.org                    TkSQL Home page - My GPL work

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Thu, 10 Nov 2005, Sandro Dentella wrote:

> On Wed, Nov 09, 2005 at 08:57:45PM -0500, Adam Rosi-Kessel wrote:
> > I'm troubeshooting an issue of accessing a VPN through NAT. Right now the
> > problem can be reduced to the following question:
> >
> > Under what conditions would inbound packets not be routing through the nat
> > PREROUTING chain?
>
> That's a problem that puzzles me too.

Packets which cannot be associated with any existing connection
known by the conntrack subsystem will traverse the NAT table.

If a packet is related to any connection, which can mean:

- the packet belongs to a connection
- it is an ICMP error packet about a connection
- it is a packet of a channel (like FTP data), which can be
  associated to a connection by an appropriate helper module

then that packet won't enter the NAT table.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

On Thu, Nov 10, 2005 at 17:44:16 CET, Jozsef Kadlecsik wrote:

> > > Under what conditions would inbound packets not be routing through
> > > the nat PREROUTING chain?
> > That's a problem that puzzles me too.
> Packets which cannot be associated with any existing connection
> known by the conntrack subsystem will traverse the NAT table.
> If a packet is related to any connection, which can mean:
> - the packet belongs to a connection
> - it is an ICMP error packet about a connection
> - it is a packet of a channel (like FTP data), which can be
>   associated to a connection by an appropriate helper module
> then that packet won't enter the NAT table.

The packets in question:

  - do show up in tcpdump (so they're at least passing by the network card)
  - do show up if logged in the mangle PREROUTING table (so iptables at
    least knows about them)
  - are UDP port 500 packets -- so that rules out the latter two options
    above, right? They are not ICMP error packets, and they are not 
    packets recognized by a channel like FTP data. I have no conntrack
    module loaded other than the main one and the FTP one.
  - do not show up in /proc/net/ip_conntrack. There in fact are no
    inbound entries at all in /proc/net/ip_conntrack for the IP address of the
    remote server or for any traffic on port 500 at all.

Yet, they do not enter the nat PREROUTING table.

I even added a raw table and a NOTRACK destination to packets travelling
on port 500 to every chain in the raw table. Still, the packets do not
show up in nat PREROUTING.

Any suggestions for how to figure out why they're not getting to nat
PREROUTING?  Or are they perhaps being tracked in a way that I am not
noticing?
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Mon, 14 Nov 2005, Adam Rosi-Kessel wrote:

> On Thu, Nov 10, 2005 at 17:44:16 CET, Jozsef Kadlecsik wrote:
>
> > > > Under what conditions would inbound packets not be routing through
> > > > the nat PREROUTING chain?
> > > That's a problem that puzzles me too.
> > Packets which cannot be associated with any existing connection
> > known by the conntrack subsystem will traverse the NAT table.
> > If a packet is related to any connection, which can mean:
> > - the packet belongs to a connection
> > - it is an ICMP error packet about a connection
> > - it is a packet of a channel (like FTP data), which can be
> >   associated to a connection by an appropriate helper module
> > then that packet won't enter the NAT table.
>
> The packets in question:
>
>   - do show up in tcpdump (so they're at least passing by the network card)
>   - do show up if logged in the mangle PREROUTING table (so iptables at
>     least knows about them)

That means they passed conntrack and was not dropped as invalid. Good.

>   - are UDP port 500 packets -- so that rules out the latter two options
>     above, right? They are not ICMP error packets, and they are not
>     packets recognized by a channel like FTP data. I have no conntrack
>     module loaded other than the main one and the FTP one.
>   - do not show up in /proc/net/ip_conntrack. There in fact are no
>     inbound entries at all in /proc/net/ip_conntrack for the IP address of the
>     remote server or for any traffic on port 500 at all.

Double-check it ;-).

> Yet, they do not enter the nat PREROUTING table.
>
> I even added a raw table and a NOTRACK destination to packets travelling
> on port 500 to every chain in the raw table. Still, the packets do not
> show up in nat PREROUTING.

If you add a NOTRACK rule for the packets in the raw table, then conntrack
will skip those packets and in consequence NAT will skip them as well: NAT
is built on top of conntrack.

> Any suggestions for how to figure out why they're not getting to nat
> PREROUTING?  Or are they perhaps being tracked in a way that I am not
> noticing?

Remove the NOTRACK rules from the raw table. Double check the mangle table
for possible matching DROP targets, including the default policies. Do the
same with the nat table as well.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 2454 bytes --]

Jozsef Kadlecsik wrote:
>>   - do not show up in /proc/net/ip_conntrack. There in fact are no
>>     inbound entries at all in /proc/net/ip_conntrack for the IP address of the
>>     remote server or for any traffic on port 500 at all.
> Double-check it ;-).

Okay--would it be sufficient to grep on port 500? conntrack tracks by port,
right? So as long as there are *no* entries in ip_conntrack for port 500 at
any point while I'm trying to make this connection, doesn't it mean that
conntrack isn't handling the packets?

>> I even added a raw table and a NOTRACK destination to packets travelling
>> on port 500 to every chain in the raw table. Still, the packets do not
>> show up in nat PREROUTING.
> If you add a NOTRACK rule for the packets in the raw table, then conntrack
> will skip those packets and in consequence NAT will skip them as well: NAT
> is built on top of conntrack.

Ah, okay, I didn't understand that.

>> Any suggestions for how to figure out why they're not getting to nat
>> PREROUTING?  Or are they perhaps being tracked in a way that I am not
>> noticing?
> Remove the NOTRACK rules from the raw table. Double check the mangle table
> for possible matching DROP targets, including the default policies. Do the
> same with the nat table as well.

The NOTRACK rules were just one test--I am not using them generally. So,
yes, they are removed.

As far as drop targets: for testing purposes, I am clearing all of the
tables of all rules, setting the default policy for all rules as ACCEPT, and
then just trying to see these inbound packets in the nat PREROUTING table.
So I am certain they are not being DROPped along the way.

The simplest possible test involves only one or two rules--SNAT'ing outbound
packets to the WAN IP, and then (attempting to) DNAT inbound packets on port
500 to the LAN IP of the proper machine. But because the inbound packets
don't enter the nat table, they never get DNAT'ed; and they are also not
routed to the proper machine with conntrack either.

I should also mention that all other NAT stuff works fine with this setup,
including FTP (passive and active), web browsing, etc..  I am also able to
DNAT inbound ssh connections to another internal machine.  It's only these
inbound udp port 500 that are somehow not entering the nat table or showing
up anywhere other than in mangle PREROUTING.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Mon, 14 Nov 2005, Adam Rosi-Kessel wrote:

> Okay--would it be sufficient to grep on port 500? conntrack tracks by port,
> right? So as long as there are *no* entries in ip_conntrack for port 500 at
> any point while I'm trying to make this connection, doesn't it mean that
> conntrack isn't handling the packets?

Yes. And if conntrack does not handle them, NAT won't do either.

> I should also mention that all other NAT stuff works fine with this setup,
> including FTP (passive and active), web browsing, etc..  I am also able to
> DNAT inbound ssh connections to another internal machine.  It's only these
> inbound udp port 500 that are somehow not entering the nat table or showing
> up anywhere other than in mangle PREROUTING.

Try to enable internal logging by

echo 255 > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid

and load in ipt_LOG or ipt_ULOG as logging engines.

Add a logging rule to the very end of the PREROUTING chain in the mangle
table:

iptables -t mangle -A PREROUTING -m state --state INVALID -j LOG \
	--log-prefix "INVALID packet: "

and let's see the result.

I suspect the UDP packets in question are marked as INVALID for some
reason.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 1252 bytes --]

Jozsef Kadlecsik wrote:
>> Okay--would it be sufficient to grep on port 500? conntrack tracks by port,
>> right? So as long as there are *no* entries in ip_conntrack for port 500 at
>> any point while I'm trying to make this connection, doesn't it mean that
>> conntrack isn't handling the packets?
> Yes. And if conntrack does not handle them, NAT won't do either.

Actually, now that I've *triple*-checked, I got a different result. I'm not
sure why this wasn't happening before, or if I was misreading the
ip_conntrack table, but now I *am* seeing the connection in question on port
500.  It is marked as ASSURED which I understand to mean that two way
traffic has occurred.

So, setting aside the question of why I wasn't seeing that before, shouldn't
I be able to see the incoming packets as they are routed to the internal
client machine, even if they are tracked connections?  When I watch the
inward-facing interface with tcpdump, I don't see any of these packets
getting routed to that machine, although I do see the outbound packets.

Does that mean that something is going wrong with the connection tracking,
or that I'm misunderstanding tcpdump, or something else?
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Tue, 15 Nov 2005, Adam Rosi-Kessel wrote:

> Jozsef Kadlecsik wrote:
> >> Okay--would it be sufficient to grep on port 500? conntrack tracks by port,
> >> right? So as long as there are *no* entries in ip_conntrack for port 500 at
> >> any point while I'm trying to make this connection, doesn't it mean that
> >> conntrack isn't handling the packets?
> > Yes. And if conntrack does not handle them, NAT won't do either.
>
> Actually, now that I've *triple*-checked, I got a different result. I'm not
> sure why this wasn't happening before, or if I was misreading the
> ip_conntrack table, but now I *am* seeing the connection in question on port
> 500.  It is marked as ASSURED which I understand to mean that two way
> traffic has occurred.

Yes. Do you see the effect of the NAT table on the conntrack entry?

> So, setting aside the question of why I wasn't seeing that before, shouldn't
> I be able to see the incoming packets as they are routed to the internal
> client machine, even if they are tracked connections?  When I watch the
> inward-facing interface with tcpdump, I don't see any of these packets
> getting routed to that machine, although I do see the outbound packets.

I don't clearly understand you here. It is always best to run tcpdump on
both interfaces so that one can compare what packets are routed properly
and how they were mangled/NAT-ed by the firewall. If some packets are
missing from either side then that's a clear sign that those packets were
dropped by either a matching rule/policy or by the system itself.

Did the logging produce anything?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

Jozsef: By the way, I'm unable to email you directly either from my
personal account or from gmail. I assume that is by design, but I thought
I'd mention it just in case.

On Tue, Nov 15, 2005 at 03:00:18PM +0100, Jozsef Kadlecsik wrote:
> > So, setting aside the question of why I wasn't seeing that before, shouldn't
> > I be able to see the incoming packets as they are routed to the internal
> > client machine, even if they are tracked connections?  When I watch the
> > inward-facing interface with tcpdump, I don't see any of these packets
> > getting routed to that machine, although I do see the outbound packets.
> I don't clearly understand you here. It is always best to run tcpdump on
> both interfaces so that one can compare what packets are routed properly
> and how they were mangled/NAT-ed by the firewall. If some packets are
> missing from either side then that's a clear sign that those packets were
> dropped by either a matching rule/policy or by the system itself.
> Did the logging produce anything?

No luck with logging, but I don't seem to have
/proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid 
even with ipt_LOG or ipt_ULOG loaded.

I'm running kernel 2.6.8-2-k7 from Debian Stable, iptables v1.2.11.

I did try adding the LOG rule to mangle PREROUTING for INVALID packets
but there was no result.

Below is the output of everything I can think of that would be relevant.
As per your suggestion, I ran tcpdump on all interfaces.  What you'll
notice is that there are packets coming in that go from the internal
client to the external server, and then are remapped form the NAT box's
external IP address to the external server (on the way out); and you'll
see packets coming in from the external server to the NAT box, but they
never go out the NAT box's LAN-facing interface to the internal client.

That is:

  LOCAL_CLIENT = machine behind the NAT box in the LAN with a 192.168.* IP address
  EXTERNAL_IP_OF_NAT_BOX = machine that has two network cards, one facing
           the Internet the other facing the LAN, where this IP address is its
           external IP address
  SERVER = the VPN server I am attempting to connect to on the Internet

# iptables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination
LOG        all  --  0.0.0.0/0            0.0.0.0/0           state INVALID LOG flags 0 level 4 prefix `INVALID packet: '

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination

# grep "INVALID" /var/log/syslog

[no results after several minutes of attempting to connect]

# iptables -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
SNAT       all  --  192.168.98.0/24      0.0.0.0/0           to:EXTERNAL_IP_OF_NAT_BOX

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

# netstat-nat -n | grep 500
udp   LOCAL_CLIENT:500              SERVER:500            ASSURED

# tcpdump -i any 'udp port 500'
18:40:06.977738 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:40:06.977836 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:40:07.620295 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:40:15.154788 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:40:24.305185 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:40:31.012740 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:40:31.012812 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:40:41.588975 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:40:59.054904 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:40:59.055032 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:41:21.622936 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:41:21.623044 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:41:21.974515 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:41:28.358127 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:41:31.103115 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:41:31.103171 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:41:37.508037 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:41:44.122683 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:41:44.122824 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:41:55.808152 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:42:00.146797 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:42:00.146908 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:42:19.175413 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:42:19.175524 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:42:41.208547 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:42:41.208661 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:42:41.560743 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:42:45.930721 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:42:55.085313 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
18:43:06.247160 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
18:43:06.247272 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
18:43:14.093473 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

On Tue, Nov 15, 2005 at 06:53:19PM -0500, Adam Rosi-Kessel wrote:
> > > So, setting aside the question of why I wasn't seeing that before, shouldn't
> > > I be able to see the incoming packets as they are routed to the internal
> > > client machine, even if they are tracked connections?  When I watch the
> > > inward-facing interface with tcpdump, I don't see any of these packets
> > > getting routed to that machine, although I do see the outbound packets.
> > I don't clearly understand you here. It is always best to run tcpdump on
> > both interfaces so that one can compare what packets are routed properly
> > and how they were mangled/NAT-ed by the firewall. If some packets are
> > missing from either side then that's a clear sign that those packets were
> > dropped by either a matching rule/policy or by the system itself.
> > Did the logging produce anything?

I should probably also mention that the NAT box has two external IP
addresses, both on eth0 (eth0 and eth0:1), although I don't think this
should affect anything, maybe there's something I don't know. All
outbound traffic from the LAN is SNAT'ed to the eth0:1 external IP
address, and the VPN traffic I'm seeing is coming back into that same IP
address.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

On Tue, Nov 15, 2005 at 06:57:45PM -0500, Adam Rosi-Kessel wrote:
> On Tue, Nov 15, 2005 at 06:53:19PM -0500, Adam Rosi-Kessel wrote:
> > > > So, setting aside the question of why I wasn't seeing that before, shouldn't
> > > > I be able to see the incoming packets as they are routed to the internal
> > > > client machine, even if they are tracked connections?  When I watch the
> > > > inward-facing interface with tcpdump, I don't see any of these packets
> > > > getting routed to that machine, although I do see the outbound packets.
> > > I don't clearly understand you here. It is always best to run tcpdump on
> > > both interfaces so that one can compare what packets are routed properly
> > > and how they were mangled/NAT-ed by the firewall. If some packets are
> > > missing from either side then that's a clear sign that those packets were
> > > dropped by either a matching rule/policy or by the system itself.
> > > Did the logging produce anything?
> I should probably also mention that the NAT box has two external IP
> addresses, both on eth0 (eth0 and eth0:1), although I don't think this
> should affect anything, maybe there's something I don't know. All
> outbound traffic from the LAN is SNAT'ed to the eth0:1 external IP
> address, and the VPN traffic I'm seeing is coming back into that same IP
> address.

Actually, it's probably even not worth mentioning. If I bring down eth0:1
and just do everything through one IP address on eth0, I get the same
results as before.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Tue, 15 Nov 2005, Adam Rosi-Kessel wrote:

> Jozsef: By the way, I'm unable to email you directly either from my
> personal account or from gmail. I assume that is by design, but I thought
> I'd mention it just in case.

*.dsl.speakeasy.net is blacklisted at our campus.

> No luck with logging, but I don't seem to have
> /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
> even with ipt_LOG or ipt_ULOG loaded.
>
> I'm running kernel 2.6.8-2-k7 from Debian Stable, iptables v1.2.11.

Try to upgrade to a newer kernel.

> I did try adding the LOG rule to mangle PREROUTING for INVALID packets
> but there was no result.

That means the packets are not INVALID then. Drop the INVALID state
matching part from the logging rule and repeat the test.

> Below is the output of everything I can think of that would be relevant.
[...]
> # netstat-nat -n | grep 500
> udp   LOCAL_CLIENT:500              SERVER:500            ASSURED

The connection is in the conntrack table and the NAT rule was applied.
Good. But next time please send the output of
'grep port=500 /proc/net/ip_conntrack' instead.

> # tcpdump -i any 'udp port 500'
> 18:40:06.977738 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
> 18:40:06.977836 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg

Packet natted and sent out.

> 18:40:07.620295 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
> 18:40:15.154788 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg

Server responds but it seems the firewall does not forward the packet.

I have no better idea than repeating the test with general logging in the
mangle table. You mentioned the raw table earlier. What is the output of
'iptables -t raw -L'? Do you know what (other) patches from
patch-o-matic(-ng) was applied in your kernel?

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

On Wed, Nov 16, 2005 at 10:42:38AM +0100, Jozsef Kadlecsik wrote:
> > Jozsef: By the way, I'm unable to email you directly either from my
> > personal account or from gmail. I assume that is by design, but I thought
> > I'd mention it just in case.
> *.dsl.speakeasy.net is blacklisted at our campus.

As is, apparently, gmail? Pretty draconian blacklist!

> > No luck with logging, but I don't seem to have
> > /proc/sys/net/ipv4/netfilter/ip_conntrack_log_invalid
> > even with ipt_LOG or ipt_ULOG loaded.
> > I'm running kernel 2.6.8-2-k7 from Debian Stable, iptables v1.2.11.
> Try to upgrade to a newer kernel.

I guess I'd do this as a last resort. I'm trying to keep the system in
question on Debian Stable if possible. Is there reason to think a kernel
upgrade would just fix it?

> > I did try adding the LOG rule to mangle PREROUTING for INVALID packets
> > but there was no result.
> That means the packets are not INVALID then. Drop the INVALID state
> matching part from the logging rule and repeat the test.

Okay, I've done that below. I added logging rules for mangle PREROUTING
and POSTROUTING with descriptive log prefixes as will be clear below.

> > Below is the output of everything I can think of that would be relevant.
> [...]
> > # netstat-nat -n | grep 500
> > udp   LOCAL_CLIENT:500              SERVER:500            ASSURED
> The connection is in the conntrack table and the NAT rule was applied.
> Good. But next time please send the output of
> 'grep port=500 /proc/net/ip_conntrack' instead.

Okay, I've done that below as well.

> > # tcpdump -i any 'udp port 500'
> > 18:40:06.977738 IP LOCAL_CLIENT.500 > SERVER.500: isakmp: phase 1 I agg
> > 18:40:06.977836 IP EXTERNAL_IP_OF_NAT_BOX.500 > SERVER.500: isakmp: phase 1 ? agg
> Packet natted and sent out.
> > 18:40:07.620295 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
> > 18:40:15.154788 IP SERVER.500 > EXTERNAL_IP_OF_NAT_BOX.500: isakmp: phase 1 R agg
> Server responds but it seems the firewall does not forward the packet.

Right. It's just somewhere disappearing after getting into the mangle
PREROUTING table from the Internet and not getting routed properly to the
internal box. Yet, other NAT connections work fine.

> I have no better idea than repeating the test with general logging in the
> mangle table. You mentioned the raw table earlier. What is the output of
> 'iptables -t raw -L'? Do you know what (other) patches from
> patch-o-matic(-ng) was applied in your kernel?

The raw table is empty, see below.

I don't think the Debian stock kernel has any patch-o-matic patches
applied. I think the only netfilter/iptables patch that is applied is a
security hole that lead to memory corruption:

http://svn.debian.org/wsvn/kernel/patch-tracking/netfilter-NAT-memory-corruption.dpatch?op=file&rev=0&sc=0

Here are my logs below--showing everything as you suggested. IP and MAC
addresses are replaced with descriptive names, but otherwise this is
exactly what appears in my logs.

# grep 500 /proc/net/ip_conntrack
udp      17 163 src=LOCAL_CLIENT dst=SERVER sport=500 dport=500 src=SERVER dst=EXTERNAL_IP_OF_NAT_BOX sport=500 dport=500 [ASSURED] use=1 

# grep Mangle /var/log/syslog | sed "s/^.*kernel: //g"

Mangle Prerouting:IN=eth1 OUT= MAC=ETH1_MAC SRC=LOCAL_CLIENT DST=SERVER LEN=1584 TOS=0x00 PREC=0x00 TTL=128 ID=50876 PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Postrouting:IN= OUT=eth0 SRC=LOCAL_CLIENT DST=SERVER LEN=1500 TOS=0x00 PREC=0x00 TTL=127 ID=50876 MF PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Prerouting:IN=eth0 OUT= MAC=ETH0_MAC SRC=SERVER DST=EXTERNAL_IP_OF_NAT_BOX LEN=2288 TOS=0x00 PREC=0x00 TTL=55 ID=63908 PROTO=UDP SPT=500 DPT=500 LEN=2268 

Mangle Prerouting:IN=eth0 OUT= MAC=ETH0_MAC SRC=SERVER DST=EXTERNAL_IP_OF_NAT_BOX LEN=2288 TOS=0x00 PREC=0x00 TTL=55 ID=63910 PROTO=UDP SPT=500 DPT=500 LEN=2268 

Mangle Prerouting:IN=eth1 OUT= MAC=ETH1_MAC SRC=LOCAL_CLIENT DST=SERVER LEN=1584 TOS=0x00 PREC=0x00 TTL=128 ID=50908 PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Postrouting:IN= OUT=eth0 SRC=LOCAL_CLIENT DST=SERVER LEN=1500 TOS=0x00 PREC=0x00 TTL=127 ID=50908 MF PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Prerouting:IN=eth0 OUT= MAC=ETH0_MAC SRC=SERVER DST=EXTERNAL_IP_OF_NAT_BOX LEN=2288 TOS=0x00 PREC=0x00 TTL=55 ID=63912 PROTO=UDP SPT=500 DPT=500 LEN=2268 

Mangle Prerouting:IN=eth1 OUT= MAC=ETH1_MAC SRC=LOCAL_CLIENT DST=SERVER LEN=1584 TOS=0x00 PREC=0x00 TTL=128 ID=50933 PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Postrouting:IN= OUT=eth0 SRC=LOCAL_CLIENT DST=SERVER LEN=1500 TOS=0x00 PREC=0x00 TTL=127 ID=50933 MF PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Prerouting:IN=eth0 OUT= MAC=ETH0_MAC SRC=SERVER DST=EXTERNAL_IP_OF_NAT_BOX LEN=2288 TOS=0x00 PREC=0x00 TTL=55 ID=63916 PROTO=UDP SPT=500 DPT=500 LEN=2268 

Mangle Prerouting:IN=eth1 OUT= MAC=ETH1_MAC SRC=LOCAL_CLIENT DST=SERVER LEN=1584 TOS=0x00 PREC=0x00 TTL=128 ID=51082 PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Postrouting:IN= OUT=eth0 SRC=LOCAL_CLIENT DST=SERVER LEN=1500 TOS=0x00 PREC=0x00 TTL=127 ID=51082 MF PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Prerouting:IN=eth1 OUT= MAC=ETH1_MAC SRC=LOCAL_CLIENT DST=SERVER LEN=1584 TOS=0x00 PREC=0x00 TTL=128 ID=51190 PROTO=UDP SPT=500 DPT=500 LEN=1564 

Mangle Postrouting:IN= OUT=eth0 SRC=LOCAL_CLIENT DST=SERVER LEN=1500 TOS=0x00 PREC=0x00 TTL=127 ID=51190 MF PROTO=UDP SPT=500 DPT=500 LEN=1564 


# iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
SNAT       all  --  192.168.98.0/24      0.0.0.0/0           to:EXTERNAL_IP_OF_NAT_BOX 

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         


# iptables -t mangle -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         
LOG        udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:500 LOG flags 0 level 4 prefix `Mangle Prerouting:' 

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination         
LOG        udp  --  0.0.0.0/0            0.0.0.0/0           udp dpt:500 LOG flags 0 level 4 prefix `Mangle Postrouting:' 


# iptables -t filter -L -n
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         


# iptables -t raw -L -n
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Thu, 17 Nov 2005, Adam Rosi-Kessel wrote:

> > > I'm running kernel 2.6.8-2-k7 from Debian Stable, iptables v1.2.11.
> > Try to upgrade to a newer kernel.
>
> I guess I'd do this as a last resort. I'm trying to keep the system in
> question on Debian Stable if possible. Is there reason to think a kernel
> upgrade would just fix it?

Just as a last resort. But the problem is so strange, I can't recall any
patch related to such a behaviour.

> Here are my logs below--showing everything as you suggested. IP and MAC
> addresses are replaced with descriptive names, but otherwise this is
> exactly what appears in my logs.

> # grep 500 /proc/net/ip_conntrack
> udp      17 163 src=LOCAL_CLIENT dst=SERVER sport=500 dport=500 src=SERVER dst=EXTERNAL_IP_OF_NAT_BOX sport=500 dport=500 [ASSURED] use=1
>
> # grep Mangle /var/log/syslog | sed "s/^.*kernel: //g"
>
> Mangle Prerouting:IN=eth1 OUT= MAC=ETH1_MAC SRC=LOCAL_CLIENT DST=SERVER LEN=1584 TOS=0x00 PREC=0x00 TTL=128 ID=50876 PROTO=UDP SPT=500 DPT=500 LEN=1564
>
> Mangle Postrouting:IN= OUT=eth0 SRC=LOCAL_CLIENT DST=SERVER LEN=1500 TOS=0x00 PREC=0x00 TTL=127 ID=50876 MF PROTO=UDP SPT=500 DPT=500 LEN=1564
>
> Mangle Prerouting:IN=eth0 OUT= MAC=ETH0_MAC SRC=SERVER DST=EXTERNAL_IP_OF_NAT_BOX LEN=2288 TOS=0x00 PREC=0x00 TTL=55 ID=63908 PROTO=UDP SPT=500 DPT=500 LEN=2268

So at least we know the packets are seen by the mangle table. As you tried
previously to match them with the 'INVALID' state and it proved to be
false, they are valid packets according to the conntrack. The filter table
is empty, so we can point our finger to nat alone.

The hard way remained: recompile the kernel with netfilter debugging
enabled. Then repeat the connection attempts with the new kernel booted
up and let's see what shows up in the kernel log.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 1511 bytes --]

Jozsef Kadlecsik wrote:
>>> Try to upgrade to a newer kernel.
>> I guess I'd do this as a last resort. I'm trying to keep the system in
>> question on Debian Stable if possible. Is there reason to think a kernel
>> upgrade would just fix it?
> Just as a last resort. But the problem is so strange, I can't recall any
> patch related to such a behaviour.

Unfortunately, I just tried a kernel upgrade (which pulled in a new libc6)
which totally hosed my RAID/LVM. It took me the better part of the evening
to get the RAID working again on the old kernel, so now I'm quite wary of
further kernel upgrades until/unless I figure out what happened there.

Hopefully there's some way to get this working with the existing kernel.

> So at least we know the packets are seen by the mangle table. As you tried
> previously to match them with the 'INVALID' state and it proved to be
> false, they are valid packets according to the conntrack. The filter table
> is empty, so we can point our finger to nat alone.

So are there no diagnostics, absent rebuilding with netfilters debugging on,
for tracing a packet in between mangle PREROUTING and nat PREROUTING? When
exactly does the packet get connection tracked? In between mangle and nat,
or at the start of nat but before entering the PREROUTING chain?

And how is it that conn_track thinks the connection is ASSURED when there
actually hasn't been a single packet forwarded from the external server back
to the internal client?

Adam


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Fri, 18 Nov 2005, Adam Rosi-Kessel wrote:

> > Just as a last resort. But the problem is so strange, I can't recall any
> > patch related to such a behaviour.
>
> Unfortunately, I just tried a kernel upgrade (which pulled in a new libc6)
> which totally hosed my RAID/LVM. It took me the better part of the evening
> to get the RAID working again on the old kernel, so now I'm quite wary of
> further kernel upgrades until/unless I figure out what happened there.

That can happen, unfortunately. It'd be good to know the kernel version,
however, which caused the problem with RAID/LVM.

> So are there no diagnostics, absent rebuilding with netfilters debugging on,
> for tracing a packet in between mangle PREROUTING and nat PREROUTING?

Without further aid, no.

> When exactly does the packet get connection tracked? In between mangle
> and nat, or at the start of nat but before entering the PREROUTING
> chain?

The order is raw table, connection tracking, mangle, nat and filter
tables.

> And how is it that conn_track thinks the connection is ASSURED when there
> actually hasn't been a single packet forwarded from the external server back
> to the internal client?

The answer is in above. Conntrack saw the packet as did the mangle table.
Something strange happens in the nat table and without further
informations nothing much can be said.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 1618 bytes --]

Jozsef Kadlecsik wrote:
>>> Just as a last resort. But the problem is so strange, I can't recall any
>>> patch related to such a behaviour.
>> Unfortunately, I just tried a kernel upgrade (which pulled in a new libc6)
>> which totally hosed my RAID/LVM. It took me the better part of the evening
>> to get the RAID working again on the old kernel, so now I'm quite wary of
>> further kernel upgrades until/unless I figure out what happened there.
> That can happen, unfortunately. It'd be good to know the kernel version,
> however, which caused the problem with RAID/LVM.

I was using 2.6.8 from Debian stable; the upgrade to 2.6.14 hosed the LVM.
RAID actually was fine according to /proc/mdstat, but LVM was segfaulting on
vgscan, which as far as I could tell was more of a sympton than the real
problem. But I couldn't figure it out, and when I returned to 2.6.8, it
still didn't work. After some combination of bringing the RAID up and down,
flushing the LVM cache file, running the vg* tools, etc., it finally came
back up, although I can't really tell you why.

>> So are there no diagnostics, absent rebuilding with netfilters debugging on,
>> for tracing a packet in between mangle PREROUTING and nat PREROUTING?
> Without further aid, no.

Okay, so I've rebuilt 2.6.8 with NETFILTER_DEBUG=Y. What would be the next
step? From what I can tell, NETFILTER_DEBUG isn't something turned on and
off in /proc/net or /proc/sys/net, it just is "on," but I'm not quite sure
what sort of debug messages we should be trying to get and how.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Jozsef Kadlecsik]

On Sun, 20 Nov 2005, Adam Rosi-Kessel wrote:

> >> So are there no diagnostics, absent rebuilding with netfilters debugging on,
> >> for tracing a packet in between mangle PREROUTING and nat PREROUTING?
> > Without further aid, no.
>
> Okay, so I've rebuilt 2.6.8 with NETFILTER_DEBUG=Y. What would be the next
> step? From what I can tell, NETFILTER_DEBUG isn't something turned on and
> off in /proc/net or /proc/sys/net, it just is "on," but I'm not quite sure
> what sort of debug messages we should be trying to get and how.

You should get debug messages logged with the 'kernel' syslog facility.
Look at /var/log/kern.log.

I'm going to try to emulate your problem with nfsim using your setup and
the tcpdump results. Probably that can help.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlec@sunserv.kfki.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : KFKI Research Institute for Particle and Nuclear Physics
          H-1525 Budapest 114, POB. 49, Hungary

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

On Thu, Nov 24, 2005 at 12:00:17PM +0100, Jozsef Kadlecsik wrote:
> > >> So are there no diagnostics, absent rebuilding with netfilters debugging on,
> > >> for tracing a packet in between mangle PREROUTING and nat PREROUTING?
> > > Without further aid, no.
> > Okay, so I've rebuilt 2.6.8 with NETFILTER_DEBUG=Y. What would be the next
> > step? From what I can tell, NETFILTER_DEBUG isn't something turned on and
> > off in /proc/net or /proc/sys/net, it just is "on," but I'm not quite sure
> > what sort of debug messages we should be trying to get and how.
> You should get debug messages logged with the 'kernel' syslog facility.
> Look at /var/log/kern.log.

Strangely, all that is appearing in /var/log/kern.log are my LOG targets,
in other words, exactly what I was getting with NETFILTER_DEBUG=N.

# grep NETFILTER_DEBUG /boot/config-`uname -r`
CONFIG_NETFILTER_DEBUG=y

# grep -v EEE /var/log/kern.log
<no results>

(where EEE is part of my prefix in --log-prefix for the LOG target).

> I'm going to try to emulate your problem with nfsim using your setup and
> the tcpdump results. Probably that can help.

That would be great. Thanks--
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 1275 bytes --]

On Fri, Nov 18, 2005 at 12:07:59PM +0100, Jozsef Kadlecsik wrote:
> > > > I'm running kernel 2.6.8-2-k7 from Debian Stable, iptables v1.2.11.
> > > Try to upgrade to a newer kernel.
> > I guess I'd do this as a last resort. I'm trying to keep the system in
> > question on Debian Stable if possible. Is there reason to think a kernel
> > upgrade would just fix it?
> Just as a last resort. But the problem is so strange, I can't recall any
> patch related to such a behaviour.

I've finally managed to upgrade the kernel without hosing my RAID/LVM
setup.

Unfortunately, I still have the identical problem. I'm now using
2.6.14-2-k7 from Debian unstable.

Packets go from client to nat box, then nat to server, then back from
server to nat box, but never then back to internal client.

The connection is showing up as tracked in /proc/net/ip_conntrack.

Even if I force nat encapsulation by SNATing the packets to unprivileged
ports, I get the same result (other than that the reply packets come back
on whatever unprivileged port was selected).

Does anyone have any other ideas about how to troubleshoot this problem?
I was really hoping the kernel upgrade would fix it, but unfortunately
not.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Leonardo Rodrigues Magalhães]

>
># iptables -t nat -L -n
>Chain PREROUTING (policy ACCEPT)
>target     prot opt source               destination         
>
>Chain POSTROUTING (policy ACCEPT)
>target     prot opt source               destination         
>SNAT       all  --  192.168.98.0/24      0.0.0.0/0           to:EXTERNAL_IP_OF_NAT_BOX 
>  
>

    Adam, have you tried adding a properly PREROUTING rule for this 
specific traffic ? It would be like

iptables -t nat -A PREROUTING -p udp -d EXTERNAL_IP_OF_NAT_BOT --dport 
500 -j DNAT --to LOCAL_CLIENT

    Other test I would suggest ....... have you tried some other UDP 
traffic, just like DNS and see if it works ? Have you tried some TCP 
traffic and see if it works fine ?

-- 


	Atenciosamente / Sincerily,
	Leonardo Rodrigues
	Solutti Tecnologia
	http://www.solutti.com.br

	Minha armadilha de SPAM, NÃO mandem email
	gertrudes@solutti.com.br
	My SPAMTRAP, do not email it

Re: Why would certain packets not reach nat PREROUTING chain?

[Gabriel]

On Thu, 10 Nov 2005 14:44:53 +0200, Adam Rosi-Kessel  
<adam@rosi-kessel.org> wrote:

> George Alexandru Dragoi wrote:
>> Better use mangle table, all packets traverse it. And
doesn't require
>> conntrack.
>
> But I can't do DNAT in mangle, right?

You cannot. nat and mangle are two different chains.

"Note that the DNAT target is only available within the
PREROUTING and  
OUTPUT chains in the nat table" from  
http://iptables-tutorial.frozentux.net/iptables-tutorial.html#DNATTARGET

-- 
Using Opera's revolutionary e-mail client: http://www.opera.com/mail/


		
__________________________________ 
Yahoo! FareChase: Search multiple travel sites in one click.
http://farechase.yahoo.com

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

Sandro Dentella Wrote:

> > I'm troubeshooting an issue of accessing a VPN through NAT. Right now
> > the problem can be reduced to the following question:
> > Under what conditions would inbound packets not be routing through the
> > nat PREROUTING chain?
> That's a problem that puzzles me too. Do you have fancy routing tables?
> (several different tables setup w/ iproute2).

Nope. At least for the purposes of this experiment, this is the only
thing I'm trying to do. The entire task of iptables is SNATting outbound
packet from the LAN, and then attempting to DNAT inbound packets on udp
port 500 to a specific machine with the LAN. The outbound SNAT works
fine; but the inbound packets don't ever reach the nat PREROUTING chain.

> I also have a setup in which icmp packets will not get to PREROUTING.
> My understanding is that the kernel does not understand they are destined
> for that box: could that be your situation?

I don't think that's related, although I admit I don't have a thorough
understanding of the issue... why would icmp packets matter when the
issue is inbound UDP 500 packets that are showing up in tcpdump? Wouldn't
showing up in tcpdump indicate that the kernel understands the packet is
destined for that box?
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

Re: Why would certain packets not reach nat PREROUTING chain?

[Sandro Dentella]

> > I also have a setup in which icmp packets will not get to PREROUTING.
> > My understanding is that the kernel does not understand they are destined
> > for that box: could that be your situation?
> 
> I don't think that's related, although I admit I don't have a thorough
> understanding of the issue... why would icmp packets matter when the
> issue is inbound UDP 500 packets that are showing up in tcpdump? Wouldn't

  I'm not saying they are related. I just see that in both cases we have
  packets that arrive (tcpdump shows them) but are not correctly understood
  by the kernel as being destined for the box.


> showing up in tcpdump indicate that the kernel understands the packet is
> destined for that box?

  I don't think so. A packet that is to be forwareded is not "destined for
  the box". In general my understanding is that tcpdump reads everything
  that passes on the wire, as it comes before any routing decision. It
  sees also packets that are not destined for the box.

  What I would like to understand is why a packet that tcpdump show as
  destined for the box in not understood as such.

  I have been asking this since a couple of weeks on 3 different lists
  (netfilter, lartc, netdev), but didn't get a single answer...

  sandro



-- 
Sandro Dentella  *:-)
e-mail: sandro@e-den.it 
http://www.tksql.org                    TkSQL Home page - My GPL work

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 1332 bytes --]

On Sun, Dec 18, 2005 at 032:42:43M -0500, Leonardo Rodrigues Magalhes wrote:
>     Adam, have you tried adding a properly PREROUTING rule for this 
> specific traffic ? It would be like
> iptables -t nat -A PREROUTING -p udp -d EXTERNAL_IP_OF_NAT_BOT --dport 
> 500 -j DNAT --to LOCAL_CLIENT

I have tried that--same result. In fact, I've tried creating a "tunnel"
between the internal and external client by leaving out the port
limitation, i.e.:

iptables -t nat -A PREROUTING -s SERVER_IP -j DNAT --to LOCAL_CLIENT

None of these have any effect because the packets never reach the nat
PREROUTING chain. If I put a LOG rule at the start of nat PREROUTING, I
don't see the inbound packets from the server on the Internet. (I do see
those packets in the mangle PREROUTING though).

>     Other test I would suggest ....... have you tried some other UDP 
> traffic, just like DNS and see if it works ? Have you tried some TCP 
> traffic and see if it works fine ?

Everything else seems to work fine. I have another internal box to which
certain ssh connections from the Internet go, and that's fine. I'm not
sure exactly how I'd test other UDP traffic... Certainly UDP traffic that
is FORWARDed through the NAT box (e.g., RealPlayer streams) seems fine.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org

[-- Attachment #2: Digital signature --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re[2]: Why would certain packets not reach nat PREROUTING chain?

[Marcin Krol]

Hello Adam,

Look at point 6.3 here:

http://iptables-tutorial.frozentux.net/iptables-tutorial.html

There was smth in the tutorial about only the first packet from NEW
connection hitting the chains and the rest of them being handled by
conntrack, although can't remember where exactly this was located in
tutorial.

This would explain the UDP traffic being handled in its entirety
since UDP is connectionless and conntrack.

Note that this is my hypothesis, it's not like I have tested
this thoroughly.

Regards,
Marcin



Sunday, December 18, 2005, 2:16:21 PM, you wrote:

ARK> On Sun, Dec 18, 2005 at 032:42:43M -0500, Leonardo Rodrigues Magalhes wrote:
>>     Adam, have you tried adding a properly PREROUTING rule for this 
>> specific traffic ? It would be like
>> iptables -t nat -A PREROUTING -p udp -d EXTERNAL_IP_OF_NAT_BOT --dport 
>> 500 -j DNAT --to LOCAL_CLIENT

ARK> I have tried that--same result. In fact, I've tried creating a "tunnel"
ARK> between the internal and external client by leaving out the port
ARK> limitation, i.e.:

ARK> iptables -t nat -A PREROUTING -s SERVER_IP -j DNAT --to LOCAL_CLIENT

ARK> None of these have any effect because the packets never reach the nat
ARK> PREROUTING chain. If I put a LOG rule at the start of nat PREROUTING, I
ARK> don't see the inbound packets from the server on the Internet. (I do see
ARK> those packets in the mangle PREROUTING though).

>>     Other test I would suggest ....... have you tried some other UDP 
>> traffic, just like DNS and see if it works ? Have you tried some TCP 
>> traffic and see if it works fine ?

ARK> Everything else seems to work fine. I have another internal box to which
ARK> certain ssh connections from the Internet go, and that's fine. I'm not
ARK> sure exactly how I'd test other UDP traffic... Certainly UDP traffic that
ARK> is FORWARDed through the NAT box (e.g., RealPlayer streams) seems fine.



-- 
Best regards,
 Marcin                            mailto:mark@wbp.krakow.pl

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

Marcin Krol wrote:
> Hello Adam,
> Look at point 6.3 here:
> http://iptables-tutorial.frozentux.net/iptables-tutorial.html
> There was smth in the tutorial about only the first packet from NEW
> connection hitting the chains and the rest of them being handled by
> conntrack, although can't remember where exactly this was located in
> tutorial.
> This would explain the UDP traffic being handled in its entirety
> since UDP is connectionless and conntrack.

Hi Marcin:

I'm not exactly sure what you mean by the last sentence--did you leave off a
word or two?

I think what you're saying is that the subsequent packets aren't showing up
in nat PREROUTING because they're being tracked. This makes sense--once a
connection is tracked or ASSURED the packets don't go through nat PREROUTING
anymore. And the connection in question *is* being tracked -- it shows up in
/proc/net/ip_conntrack as ASSURED.

Even though the connection is tracked, the traffic *doesn't* show up when
watching the inward facing network card with ethereal or tcpdump.  With
other tracked connections (for example, an ssh or http connection) I see all
traffic with tcpdump: coming from internal client to nat box, nat box to
external server, then back from external server to nat box, and from nat box
to internal client. With this connection, however, I see 3 out of 4 with
tcpdump -- the traffic comes back into the nat box from the outside, is
apparently properly tracked according to /proc/net/ip_conntrack, but it
never leaves again on the internal facing network card.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 250 bytes --]

Re: Why would certain packets not reach nat PREROUTING chain?

[Adam Rosi-Kessel]

I'm following up on a thread that started at
<https://lists.netfilter.org/pipermail/netfilter/2005-November/063703.html>.

The problem was trying to connect to a SonicWALL VPN through a NAT box
running iptables. Packets went from client to NAT box, then NAT box to
VPN server, then back from the VPN serve to the NAT box, but disappeared
into an abyss somewhere between mangle PREROUTING and nat PREROUTING. The
connection showed up as tracked in /proc/net/ip_conntrack, but tcpdump
showed no packets going out the inward facing ethernet card.

As it turns out, an upgrade to 2.6.15.1 seems to have totally fixed the
problem. I also went through the kernel options one-by-one and turned off
absolutely everything that was not essential for the server. So either
explanation could account for the fix.

People had suspected before that the Debian stock kernel IPSec
functionality was getting in the way; the Debian kernel does come with
IPSec modules but none of them were loaded.

In any case, I thought I'd just post this to put some finality to the
discussion. Thanks for everyone who helped earlier.
-- 
Adam Rosi-Kessel
http://adam.rosi-kessel.org