forwarded ports become "filtered" instead of "open"

forwarded ports become "filtered" instead of "open"

[Anthony DiSante]

Hello,

I have two ~identical Debian systems running iptables, and I've been having 
trouble getting them to communicate with each other fully.  Both machines 
are running SSH and Rsync servers, and one of them is running as a Condor 
master, so it has a process listening on port 9618.

Both systems have the same /etc/hosts looking like this:

	127.0.0.1 localhost localhost.localdomain
	10.0.0.12 box1.fqdn.com box1
	10.0.0.13 box2.fqdn.com box2

Having the FQDNs resolving to the private IPs seems a little screwy to me, 
but the network is set up (by someone else) so that box1.fqdn.com actually 
does properly map to that system's public IP.

Now, any system on the internet outside our LAN can access the servers on 
these systems with no problems.  From any such outside system, an nmap scan 
will show ports 22,873,9618 as "open."

The problem is that for some reason, these two systems can't talk to each 
other over these ports, and neither one can even access those ports on 
itself.  nmapping box1 from box1, or from box2, shows all three ports as 
"filtered" and indeed the servers are inaccessible.

Now here's where it gets weird, and I'm thinking the firewall rules must be 
messed up.  SSHing from either system to the other will always fail with a 
timeout, but if I do this:

	ssh from box1 to box2 [which fails]

And then this:

	ssh from box2 to box1 [which also fails]

...THEN when I SSH from box1 to box2 again, it suddenly works fine -- for a 
few minutes, after which the same situation occurs.  I know the firewall has 
some concept of a "session" or a "state" and I'm assuming that the 
back-and-forth is somehow enabling SSH to work temporarily as explained above.

I've edited /etc/narc/narc.conf like this:

	ALLOW_TCP_EXT="ssh,rsync,http,9618"
	ALLOW_TCP_LAN"ssh,rsync,http,9618"

...and when I restart the firewall it says this:

	Allow external connections on eth0 TCP ports: ssh,rsync,http,9618
	Allow LAN connections on eth0 TCP ports: ssh,http,rsync,9618

...yet this problem persists.

And when I disable the firewall, the problems goes away.

Can anyone offer some pointers here?  I imagine more of my narc.conf and/or 
iptables' output would be helpful, but rather than me attaching the whole 
thing right now, just ask if you want me to post any of that.

Thanks,
Anthony DiSante
http://encodable.com/
http://nodivisions.com/

Re: forwarded ports become "filtered" instead of "open"

[Anthony DiSante]

Jörg Harmuth wrote:
>>Can anyone offer some pointers here?  I imagine more of my narc.conf
>>and/or iptables' output would be helpful, but rather than me attaching
>>the whole thing right now, just ask if you want me to post any of that.
> 
> 
> Yes, please do "iptables-save > <filename>" and paste the content of the
> file into your posting, after changing information that must not be
> public (if any). Some other info like kernel version, interfaces (if
> more than one)and iptables version may be useful too.

I only have one network interface (not counting lo).  Here's the rest of 
that info:

# uname -a
Linux box1 2.6.8-2-686-smp #1 SMP Thu May 19 17:27:55 JST 2005 i686 GNU/Linux

# iptables --version
iptables v1.3.1

# iptables-save
# Generated by iptables-save v1.3.1 on Wed Aug 10 13:21:27 2005
*mangle
:PREROUTING ACCEPT [3810743:1593668883]
:INPUT ACCEPT [3810723:1593667323]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [5229526:6297115310]
:POSTROUTING ACCEPT [5233616:6298102388]
-A PREROUTING -d 0.0.0.0/255.0.0.0 -i eth0 -m state --state NEW -j DROP
-A PREROUTING -d 255.255.255.255 -i eth0 -m state --state NEW -j DROP
-A PREROUTING -d 224.0.0.0/240.0.0.0 -i eth0 -m state --state NEW -j DROP
-A PREROUTING -d 0.0.0.0/255.0.0.0 -i eth0 -m state --state NEW -j DROP
-A PREROUTING -d 255.255.255.255 -i eth0 -m state --state NEW -j DROP
-A PREROUTING -d 224.0.0.0/240.0.0.0 -i eth0 -m state --state NEW -j DROP
COMMIT
# Completed on Wed Aug 10 13:21:27 2005
# Generated by iptables-save v1.3.1 on Wed Aug 10 13:21:27 2005
*nat
:PREROUTING ACCEPT [253511:23814701]
:POSTROUTING ACCEPT [23451:1825798]
:OUTPUT ACCEPT [23451:1825798]
COMMIT
# Completed on Wed Aug 10 13:21:27 2005
# Generated by iptables-save v1.3.1 on Wed Aug 10 13:21:27 2005
*filter
:CUST_LOG - [0:0]
:ICMP_CHK - [0:0]
:INPUT DROP [5:272]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [60966:44974754]
:SANITY_CHK - [0:0]
:SPOOF_CHK - [0:0]
:STATE_CHK - [0:0]
:TCP_CHK - [0:0]
:UDP_CHK - [0:0]
-A CUST_LOG -s 127.0.0.0/255.0.0.0 -i eth0 -j LOG --log-prefix "SPOOF " 
--log-level 7 --log-tcp-options --log-ip-options
-A CUST_LOG -s 240.0.0.0/248.0.0.0 -i eth0 -j LOG --log-prefix "SPOOF " 
--log-level 7 --log-tcp-options --log-ip-options
-A CUST_LOG -s 248.0.0.0/248.0.0.0 -i eth0 -j LOG --log-prefix "SPOOF " 
--log-level 7 --log-tcp-options --log-ip-options
-A CUST_LOG -s 172.16.0.0/255.240.0.0 -i eth0 -j LOG --log-prefix "SPOOF " 
--log-level 7 --log-tcp-options --log-ip-options
-A CUST_LOG -s 192.168.0.0/255.255.0.0 -i eth0 -j LOG --log-prefix "SPOOF " 
--log-level 7 --log-tcp-options --log-ip-options
-A CUST_LOG -s 10.0.0.12 -i eth0 -j LOG --log-prefix "SPOOF " --log-level 7 
--log-tcp-options --log-ip-options
-A CUST_LOG -p tcp -m multiport --dports 
23,81,111,123,161,445,515,555,1234,1241,1243,1433,1494,2049,3306 -j LOG 
--log-prefix "PROBE " --log-level 7 --log-tcp-options --log-ip-options
-A CUST_LOG -p tcp -m multiport --dports 
3128,3389,5631,5632,6635,8080,9055,12345,24452,27374,27573,31337,42484 -j 
LOG --log-prefix "PROBE " --log-level 7 --log-tcp-options --log-ip-options
-A CUST_LOG -p udp -m multiport --dports 
22,161,1025,3283,5634,5882,28431,31337,31789 -j LOG --log-prefix "PROBE " 
--log-level 7 --log-ip-options
-A CUST_LOG -j DROP
-A ICMP_CHK -p icmp -m icmp --icmp-type 0 -m limit --limit 1/sec -j ACCEPT
-A ICMP_CHK -p icmp -m icmp --icmp-type 3/0 -m limit --limit 1/sec -j ACCEPT
-A ICMP_CHK -p icmp -m icmp --icmp-type 3/1 -m limit --limit 1/sec -j ACCEPT
-A ICMP_CHK -p icmp -m icmp --icmp-type 3/3 -m limit --limit 1/sec -j ACCEPT
-A ICMP_CHK -p icmp -m icmp --icmp-type 3/4 -m limit --limit 1/sec -j ACCEPT
-A ICMP_CHK -p icmp -m icmp --icmp-type 11 -m limit --limit 1/sec -j ACCEPT
-A ICMP_CHK -j DROP
-A INPUT -j SPOOF_CHK
-A INPUT -p tcp -j SANITY_CHK
-A INPUT -j STATE_CHK
-A INPUT -s 127.0.0.0/255.255.255.0 -d 127.0.0.0/255.255.255.0 -i lo -m 
state --state NEW -j ACCEPT
-A INPUT -p tcp -j TCP_CHK
-A INPUT -p udp -j UDP_CHK
-A INPUT -p icmp -j ICMP_CHK
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN -j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,PSH -j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,ACK 
-j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,PSH 
-j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST 
-j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG 
FIN,SYN,RST,PSH -j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG 
FIN,SYN,RST,ACK -j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG 
FIN,SYN,RST,PSH,ACK -j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG 
FIN,SYN,RST,PSH,ACK,URG -j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN -j DROP
-A SANITY_CHK -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP
-A SPOOF_CHK -s 127.0.0.0/255.0.0.0 -i eth0 -j CUST_LOG
-A SPOOF_CHK -s 240.0.0.0/248.0.0.0 -i eth0 -j CUST_LOG
-A SPOOF_CHK -s 248.0.0.0/248.0.0.0 -i eth0 -j CUST_LOG
-A SPOOF_CHK -s 172.16.0.0/255.240.0.0 -i eth0 -j CUST_LOG
-A SPOOF_CHK -s 192.168.0.0/255.255.0.0 -i eth0 -j CUST_LOG
-A SPOOF_CHK -s 10.0.0.12 -i eth0 -j CUST_LOG
-A STATE_CHK -m state --state RELATED,ESTABLISHED -j ACCEPT
-A STATE_CHK -m state --state INVALID -j DROP
-A STATE_CHK -p tcp -m state --state NEW -m tcp ! --tcp-flags SYN,RST,ACK 
SYN -j DROP
-A TCP_CHK -s 10.0.0.12 -d 10.0.0.12 -p tcp -m tcp --dport 3029:3038 -j ACCEPT
-A TCP_CHK -s 10.0.0.12 -d 10.0.0.12 -p tcp -m tcp --dport 5432 -j ACCEPT
-A TCP_CHK -s 10.0.0.12 -d 10.0.0.12 -p tcp -m tcp --dport 3306 -j ACCEPT
-A TCP_CHK -i eth0 -p tcp -m tcp --dport 113 --tcp-flags SYN,RST,ACK SYN -j 
REJECT --reject-with tcp-reset
-A TCP_CHK -i eth0 -p tcp -m tcp --dport 113 --tcp-flags SYN,RST,ACK SYN -j 
REJECT --reject-with tcp-reset
-A TCP_CHK -i eth0 -p tcp -m multiport --dports 22,873,80,9618 -m state 
--state NEW -m tcp --tcp-flags SYN,RST,ACK SYN -j ACCEPT
-A TCP_CHK -i eth0 -p tcp -m multiport --dports 22,80,873,9618 -m state 
--state NEW -m tcp --tcp-flags SYN,RST,ACK SYN -j ACCEPT
-A TCP_CHK -j CUST_LOG
-A UDP_CHK -j CUST_LOG
-A UDP_CHK -s 10.0.0.12 -d 10.0.0.12 -p udp -m udp --dport 5432 -j ACCEPT
-A UDP_CHK -s 10.0.0.12 -d 10.0.0.12 -p udp -m udp --dport 3306 -j ACCEPT
COMMIT
# Completed on Wed Aug 10 13:21:27 2005


Thanks,
Anthony DiSante
http://encodable.com/
http://nodivisions.com/

Re: forwarded ports become "filtered" instead of "open"

[/dev/rob0]

On Wednesday 2005-August-10 12:27, Anthony DiSante wrote:
> Jörg Harmuth wrote:
> >>Can anyone offer some pointers here?  I imagine more of my
> >> narc.conf and/or iptables' output would be helpful, but rather
> >> than me attaching the whole thing right now, just ask if you want
> >> me to post any of that.

I would point out that this "narc.conf" appears to be controlling a 
rather complex script. I cannot answer anything about it. I can look 
over your rules ... needlessly complex IMO ... and point out problems. 
But for issues with this script, consult its documentation (and source 
code) and ask elsewhere.

> # iptables-save
> # Generated by iptables-save v1.3.1 on Wed Aug 10 13:21:27 2005
> *mangle
>
> -A PREROUTING -d 0.0.0.0/255.0.0.0 -i eth0 -m state --state NEW -j
> DROP

Filtering in the nat or mangle tables is not advised. It might not even 
work as expected. I have no idea what the script writer is doing here, 
nor why s/he is doing it here in particular.

Here are the rules in INPUT:

> -A INPUT -j SPOOF_CHK
> -A INPUT -p tcp -j SANITY_CHK
> -A INPUT -j STATE_CHK
> -A INPUT -s 127.0.0.0/255.255.255.0 -d 127.0.0.0/255.255.255.0 -i lo
> -m state --state NEW -j ACCEPT
> -A INPUT -p tcp -j TCP_CHK
> -A INPUT -p udp -j UDP_CHK
> -A INPUT -p icmp -j ICMP_CHK

We'll consider SSH. That is first examined by SPOOF_CHK, then by 
SANITY_CHK, then by STATE_CHK, and finally by TCP_CHK.

> -A STATE_CHK -m state --state RELATED,ESTABLISHED -j ACCEPT

You're probably right that when you manually try in both directions, 
you're going to fool connection tracking into thinking that it's 
related.

> -A STATE_CHK -p tcp -m state --state NEW -m tcp ! --tcp-flags
> SYN,RST,ACK SYN -j DROP

I'm not sure what this does. It might be dropping your SSH depending on 
the TCP flags which are set.

> -A TCP_CHK -s 10.0.0.12 -d 10.0.0.12 -p tcp -m tcp --dport 3029:3038
> -j ACCEPT

These rules are bypassed. The last rule in SPOOF_CHK :
> -A SPOOF_CHK -s 10.0.0.12 -i eth0 -j CUST_LOG
has already dropped anything which would match. I don't understand what 
they were supposed to do anyway, both source and destination the local 
eth0 IP address?

> -A TCP_CHK -i eth0 -p tcp -m multiport --dports 22,873,80,9618 -m
> state --state NEW -m tcp --tcp-flags SYN,RST,ACK SYN -j ACCEPT

This looks like it should be accepting your SSH, but for some reason 
it's not. What is the IP you are using, the NAT'ed external IP, or the 
direct 10.0.0.12 and .13? If you're using the NAT'ed IP, the problem is 
on the router which is doing the NAT. (And I agree, that was a lousy 
way to set this up.)

> -A TCP_CHK -i eth0 -p tcp -m multiport --dports 22,80,873,9618 -m
> state --state NEW -m tcp --tcp-flags SYN,RST,ACK SYN -j ACCEPT

Same rule twice? That's strange.

Anyway, your solution is quite simple. Put an ACCEPT rule ahead of 
anything which would drop or reject the packets you want.
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header