some addresses won't route

some addresses won't route

[iptables-user]

Hi list members,

I created what I thought was a simple 3 network router which worked 
great for 4 or 5 days, but has gone bonkers.  Restarting it doesn't make 
it work correctly, neither does rebooting.  I have a hunch that 
something in a cache somewhere may have expired or one of the flags in 
the /proc tree changed but I sure don't know what.

I'm using unpatched iptables-1.2.11 on fc4 with unmodified kernel.

Box is setup as a router with a WAN, DMZ, and LAN.  WAN traffic DNAT'd 
to DMZ works. DMZ and LAN through WAN works. The problems show up in LAN 
to DMZ traffic.

 From LAN to/through DMZ ping (icmp), dns (udp and tcp), and ssh work 
fine. pop3 and smtp work, but only after a looong wait, much longer than 
a dns timeout. http works on one DMZ'd server, but on another webserver 
with 2 IPs will only connect to one of the IPs (the one that the 
webserver is NOT listening to, but works correctly for WAN traffic).

Sniffing with tcpdump on DMZ for pop3 or smtp traffic shows syn/ack/ack 
followed by a minutes long wait.  Sniffing for http on DMZ shows correct 
traffic for D.M.Z.12, but for D.M.Z.11 never shows up on the DMZ 
interface (11 and 12 are on the same dev).  Switching the order the 
addresses are added to the interface has no effect.

All nics on all machines are brought up with "ifconfig ethX up" and 
addresses are attached using "ip addr add a.b.c.d/nm dev ethX".  Default 
routes are created, and on the router /proc/sys/net/ipv4/ip_forward is 
set to "1".

On all machines ifconfig, ip addr show, and route display expected results.

The puzzler is that it worked so well for 4 or 5 days.

Here is the iptables rule set which gets loaded using iptables-restore.

##########  VERY BASIC 3-LEGGED FIREWALL/ROUTER  ###########
#
# [eth0] LAN is L.A.N.1   /24 (private)
# [eth1] WAN is W.A.N.1-5 /29 (dsl to internet)
# [eth2] DMZ is D.M.Z.1   /24 (servers)
#
*nat
# remember: only NEW connections go through PREROUTING
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:GO_1 - [0:0]
:GO_2 - [0:0]
:GO_3 - [0:0]
:GO_4 - [0:0]
:GO_5 - [0:0]
# filtering belongs in filter table...
-A PREROUTING -p icmp -j RETURN
# divvy ip's into chains; it's faster
-A PREROUTING -d W.A.N.1 -j GO_1
-A PREROUTING -d W.A.N.2 -j GO_2
-A PREROUTING -d W.A.N.3 -j GO_3
-A PREROUTING -d W.A.N.4 -j GO_4
-A PREROUTING -d W.A.N.5 -j GO_5
# round-robin source ip's make visual log inspection easier for me
-A POSTROUTING -o WAN -j SNAT --to-source W.A.N.1-W.A.N.5
# DNAT maps
#  eg: -I GO_3 -p tcp --dport 80 -j DNAT --to-destination D.M.Z.100
#    would map http://W.A.N.3 to http://D.M.Z.100
-A GO_1 -j DROP
-A GO_2 -j DROP
-A GO_3 -j DROP
-A GO_4 -j DROP
-A GO_5 -j DROP
COMMIT
#
*filter
:INPUT   DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]
# ok, so i'm an idiot. i like to talk to myself
-A INPUT -i lo -j ACCEPT
# wan is shy
-A INPUT -i WAN -p icmp -j DROP
# but the rest of us aren't :)
-A INPUT -p icmp -j ACCEPT
# allow router administration from lan
-A INPUT -s L.A.N.0/255.255.255.0 -d L.A.N.1 -p tcp -m tcp --dport 22 -j 
ACCEPT
#
# let it route...
-A FORWARD -o DMZ -j ACCEPT
# let it route...
-A FORWARD -o WAN -j ACCEPT
# lan offers no services
-A FORWARD -o LAN -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT

Any ideas?  I'm at my (half) wits end.

Thanks for any help,
San Jose Mike

Re: some addresses won't route

[/dev/rob0]

On Sunday 18 September 2005 14:14, iptables-user wrote:
> I created what I thought was a simple 3 network router which worked
> great for 4 or 5 days, but has gone bonkers.  Restarting it doesn't
> make it work correctly, neither does rebooting.  I have a hunch that
> something in a cache somewhere may have expired or one of the flags
> in the /proc tree changed but I sure don't know what.

The /proc tree is dynamic; it does not persist across reboots. If 
something is changing it, it's being done by the OS startup scripts.

> I'm using unpatched iptables-1.2.11 on fc4 with unmodified kernel.

Very close to 100% chance that it will work when set up properly. A 
common error is to omit something important from a custom kernel, but 
distro kernels tend not to have those problems.

> Box is setup as a router with a WAN, DMZ, and LAN.  WAN traffic
> DNAT'd to DMZ works. DMZ and LAN through WAN works. The problems show
> up in LAN to DMZ traffic.

Incomplete description. Does the LAN and the DMZ have different 
netblocks? What IP addresses are being accessed by LAN clients? Are you 
doing the DNAT for those?

>  From LAN to/through DMZ ping (icmp), dns (udp and tcp), and ssh work
> fine. pop3 and smtp work, but only after a looong wait, much longer
> than a dns timeout. http works on one DMZ'd server, but on another
> webserver with 2 IPs will only connect to one of the IPs (the one
> that the webserver is NOT listening to, but works correctly for WAN
> traffic).

/me reaches for the aspirin bottle

> Sniffing with tcpdump on DMZ for pop3 or smtp traffic shows
> syn/ack/ack followed by a minutes long wait.  Sniffing for http on
> DMZ shows correct traffic for D.M.Z.12, but for D.M.Z.11 never shows
> up on the DMZ interface (11 and 12 are on the same dev).  Switching
> the order the addresses are added to the interface has no effect.
>
> All nics on all machines are brought up with "ifconfig ethX up" and
> addresses are attached using "ip addr add a.b.c.d/nm dev ethX". 
> Default routes are created, and on the router
> /proc/sys/net/ipv4/ip_forward is set to "1".
>
> On all machines ifconfig, ip addr show, and route display expected
> results.
>
> The puzzler is that it worked so well for 4 or 5 days.

More of it is puzzling to me. I think we missed part of the story.

> Here is the iptables rule set which gets loaded using
> iptables-restore.
>
> ##########  VERY BASIC 3-LEGGED FIREWALL/ROUTER  ###########
> #
> # [eth0] LAN is L.A.N.1   /24 (private)
> # [eth1] WAN is W.A.N.1-5 /29 (dsl to internet)

Your iptables box is listening on all of these WAN IP's? Is it a /28?

> # [eth2] DMZ is D.M.Z.1   /24 (servers)

Are these another RFC 1918 netblock? If so why are you obscuring the 
IP's?

> *nat
> # remember: only NEW connections go through PREROUTING
>
> :PREROUTING ACCEPT [0:0]
> :POSTROUTING ACCEPT [0:0]
> :OUTPUT ACCEPT [0:0]
> :GO_1 - [0:0]
> :GO_2 - [0:0]
> :GO_3 - [0:0]
> :GO_4 - [0:0]
> :GO_5 - [0:0]
>
> # filtering belongs in filter table...

Amen, brother, preach on!

> -A PREROUTING -p icmp -j RETURN

What does a RETURN do from a builtin chain? I don't understand what 
you're trying to do with that rule. If it works like ACCEPT, in the nat 
table that just means no NAT will be done.

> # divvy ip's into chains; it's faster
> -A PREROUTING -d W.A.N.1 -j GO_1
> -A PREROUTING -d W.A.N.2 -j GO_2
> -A PREROUTING -d W.A.N.3 -j GO_3
> -A PREROUTING -d W.A.N.4 -j GO_4
> -A PREROUTING -d W.A.N.5 -j GO_5

So each WAN IP has its own chain in nat PREROUTING. Not sure why.

> # round-robin source ip's make visual log inspection easier for me

But I don't see any logging anyway, are we missing something here?

> -A POSTROUTING -o WAN -j SNAT --to-source W.A.N.1-W.A.N.5

And I don't understand the benefit in this. I'd use a single IP for the 
SNAT'ed LAN clients and save the others for other uses.

> # DNAT maps
> #  eg: -I GO_3 -p tcp --dport 80 -j DNAT --to-destination D.M.Z.100
> #    would map http://W.A.N.3 to http://D.M.Z.100

Okay, so it's a DNAT-style DMZ using RFC 1918 addresses for the DMZ? I 
wouldn't do this. I'd use the "real" IP's in the DMZ. Use rules in 
filter FORWARD to control access from the outside. Less need for 
aspirin, and more likely to make it Just Work As Intended.

You have omitted the actual DNAT mapping rules? This is confusing. 
Please don't obfuscate your issue beyond the ability to troubleshoot 
it, if you want help with it.

> -A GO_1 -j DROP
> -A GO_2 -j DROP
> -A GO_3 -j DROP
> -A GO_4 -j DROP
> -A GO_5 -j DROP

But brother, practice what thou preacheth! You said you would not do 
filtering in the nat table, and yet, you do! Why?

You have left important parts out, but these DROP rules could be your 
culprit.

> COMMIT
> #
> *filter
>
> :INPUT   DROP [0:0]
> :FORWARD DROP [0:0]
> :OUTPUT ACCEPT [0:0]
>
> # ok, so i'm an idiot. i like to talk to myself
> -A INPUT -i lo -j ACCEPT
> # wan is shy
> -A INPUT -i WAN -p icmp -j DROP

Not a good idea to put this rule here. Some ICMP you will want to get, 
even if you block certain types.

> # but the rest of us aren't :)
> -A INPUT -p icmp -j ACCEPT
> # allow router administration from lan
> -A INPUT -s L.A.N.0/255.255.255.0 -d L.A.N.1 -p tcp -m tcp --dport 22
> -j ACCEPT

No --state RELATED,ESTABLISHED -j ACCEPT rule here? I use a "State" 
chain with this rule, and jump to it from both INPUT and FORWARD, at 
(or very near) the top of both chains. This would allow you to get the 
ICMP replies as alluded above.

> # let it route...
> -A FORWARD -o DMZ -j ACCEPT
> # let it route...
> -A FORWARD -o WAN -j ACCEPT
> # lan offers no services
> -A FORWARD -o LAN -m state --state RELATED,ESTABLISHED -j ACCEPT
> COMMIT

I think if you trace what is happening to each connection you might 
figure this out. But still, I'd recommend doing away with the DNAT DMZ 
and using routed IP's instead.
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header

Re: some addresses won't route

[wkc]

/dev/rob0 wrote:

>On Sunday 18 September 2005 14:14, iptables-user wrote:
>  
>
>>I created what I thought was a simple 3 network router which worked
>>great for 4 or 5 days, but has gone bonkers.  Restarting it doesn't
>>make it work correctly, neither does rebooting.  I have a hunch that
>>something in a cache somewhere may have expired or one of the flags
>>in the /proc tree changed but I sure don't know what.
>>    
>>
>
>The /proc tree is dynamic; it does not persist across reboots. If 
>something is changing it, it's being done by the OS startup scripts.
>
>  
>
>>I'm using unpatched iptables-1.2.11 on fc4 with unmodified kernel.
>>    
>>
>
>Very close to 100% chance that it will work when set up properly. A 
>common error is to omit something important from a custom kernel, but 
>distro kernels tend not to have those problems.
>
>  
>
>>Box is setup as a router with a WAN, DMZ, and LAN.  WAN traffic
>>DNAT'd to DMZ works. DMZ and LAN through WAN works. The problems show
>>up in LAN to DMZ traffic.
>>    
>>
>
>Incomplete description. Does the LAN and the DMZ have different 
>netblocks? What IP addresses are being accessed by LAN clients? Are you 
>doing the DNAT for those?
>
>  
>
>> From LAN to/through DMZ ping (icmp), dns (udp and tcp), and ssh work
>>fine. pop3 and smtp work, but only after a looong wait, much longer
>>than a dns timeout. http works on one DMZ'd server, but on another
>>webserver with 2 IPs will only connect to one of the IPs (the one
>>that the webserver is NOT listening to, but works correctly for WAN
>>traffic).
>>    
>>
>
>/me reaches for the aspirin bottle
>
>  
>
>>Sniffing with tcpdump on DMZ for pop3 or smtp traffic shows
>>syn/ack/ack followed by a minutes long wait.  Sniffing for http on
>>DMZ shows correct traffic for D.M.Z.12, but for D.M.Z.11 never shows
>>up on the DMZ interface (11 and 12 are on the same dev).  Switching
>>the order the addresses are added to the interface has no effect.
>>
>>All nics on all machines are brought up with "ifconfig ethX up" and
>>addresses are attached using "ip addr add a.b.c.d/nm dev ethX". 
>>Default routes are created, and on the router
>>/proc/sys/net/ipv4/ip_forward is set to "1".
>>
>>On all machines ifconfig, ip addr show, and route display expected
>>results.
>>
>>The puzzler is that it worked so well for 4 or 5 days.
>>    
>>
>
>More of it is puzzling to me. I think we missed part of the story.
>
>  
>
>>Here is the iptables rule set which gets loaded using
>>iptables-restore.
>>
>>##########  VERY BASIC 3-LEGGED FIREWALL/ROUTER  ###########
>>#
>># [eth0] LAN is L.A.N.1   /24 (private)
>># [eth1] WAN is W.A.N.1-5 /29 (dsl to internet)
>>    
>>
>
>Your iptables box is listening on all of these WAN IP's? Is it a /28?
>
>  
>
>># [eth2] DMZ is D.M.Z.1   /24 (servers)
>>    
>>
>
>Are these another RFC 1918 netblock? If so why are you obscuring the 
>IP's?
>
>  
>
>>*nat
>># remember: only NEW connections go through PREROUTING
>>
>>:PREROUTING ACCEPT [0:0]
>>:POSTROUTING ACCEPT [0:0]
>>:OUTPUT ACCEPT [0:0]
>>:GO_1 - [0:0]
>>:GO_2 - [0:0]
>>:GO_3 - [0:0]
>>:GO_4 - [0:0]
>>:GO_5 - [0:0]
>>
>># filtering belongs in filter table...
>>    
>>
>
>Amen, brother, preach on!
>
>  
>
>>-A PREROUTING -p icmp -j RETURN
>>    
>>
>
>What does a RETURN do from a builtin chain? I don't understand what 
>you're trying to do with that rule. If it works like ACCEPT, in the nat 
>table that just means no NAT will be done.
>
>  
>
>># divvy ip's into chains; it's faster
>>-A PREROUTING -d W.A.N.1 -j GO_1
>>-A PREROUTING -d W.A.N.2 -j GO_2
>>-A PREROUTING -d W.A.N.3 -j GO_3
>>-A PREROUTING -d W.A.N.4 -j GO_4
>>-A PREROUTING -d W.A.N.5 -j GO_5
>>    
>>
>
>So each WAN IP has its own chain in nat PREROUTING. Not sure why.
>
>  
>
>># round-robin source ip's make visual log inspection easier for me
>>    
>>
>
>But I don't see any logging anyway, are we missing something here?
>
>  
>
>>-A POSTROUTING -o WAN -j SNAT --to-source W.A.N.1-W.A.N.5
>>    
>>
>
>And I don't understand the benefit in this. I'd use a single IP for the 
>SNAT'ed LAN clients and save the others for other uses.
>
>  
>
>># DNAT maps
>>#  eg: -I GO_3 -p tcp --dport 80 -j DNAT --to-destination D.M.Z.100
>>#    would map http://W.A.N.3 to http://D.M.Z.100
>>    
>>
>
>Okay, so it's a DNAT-style DMZ using RFC 1918 addresses for the DMZ? I 
>wouldn't do this. I'd use the "real" IP's in the DMZ. Use rules in 
>filter FORWARD to control access from the outside. Less need for 
>aspirin, and more likely to make it Just Work As Intended.
>
>You have omitted the actual DNAT mapping rules? This is confusing. 
>Please don't obfuscate your issue beyond the ability to troubleshoot 
>it, if you want help with it.
>
>  
>
>>-A GO_1 -j DROP
>>-A GO_2 -j DROP
>>-A GO_3 -j DROP
>>-A GO_4 -j DROP
>>-A GO_5 -j DROP
>>    
>>
>
>But brother, practice what thou preacheth! You said you would not do 
>filtering in the nat table, and yet, you do! Why?
>
>You have left important parts out, but these DROP rules could be your 
>culprit.
>
>  
>
>>COMMIT
>>#
>>*filter
>>
>>:INPUT   DROP [0:0]
>>:FORWARD DROP [0:0]
>>:OUTPUT ACCEPT [0:0]
>>
>># ok, so i'm an idiot. i like to talk to myself
>>-A INPUT -i lo -j ACCEPT
>># wan is shy
>>-A INPUT -i WAN -p icmp -j DROP
>>    
>>
>
>Not a good idea to put this rule here. Some ICMP you will want to get, 
>even if you block certain types.
>
>  
>
>># but the rest of us aren't :)
>>-A INPUT -p icmp -j ACCEPT
>># allow router administration from lan
>>-A INPUT -s L.A.N.0/255.255.255.0 -d L.A.N.1 -p tcp -m tcp --dport 22
>>-j ACCEPT
>>    
>>
>
>No --state RELATED,ESTABLISHED -j ACCEPT rule here? I use a "State" 
>chain with this rule, and jump to it from both INPUT and FORWARD, at 
>(or very near) the top of both chains. This would allow you to get the 
>ICMP replies as alluded above.
>
>  
>
>># let it route...
>>-A FORWARD -o DMZ -j ACCEPT
>># let it route...
>>-A FORWARD -o WAN -j ACCEPT
>># lan offers no services
>>-A FORWARD -o LAN -m state --state RELATED,ESTABLISHED -j ACCEPT
>>COMMIT
>>    
>>
>
>I think if you trace what is happening to each connection you might 
>figure this out. But still, I'd recommend doing away with the DNAT DMZ 
>and using routed IP's instead.
>  
>
Just a thought; this sounds a bit like a name resolution problem to me. 
Have you tried using ip addresses in place of names? HTH

Cheers

-- 
Keith Clethero



System Administrator
Taranaki Sawmills Ltd.
Ph 06 7559000 x 816

Re: some addresses won't route

[iptables-user]

/dev/rob0 wrote:

Thanks for the detailed reply.  Your efforts are _very_ appreciated.

> On Sunday 18 September 2005 14:14, iptables-user wrote:
> 
>>I created what I thought was a simple 3 network router which worked
>>great for 4 or 5 days, but has gone bonkers.  Restarting it doesn't
>>make it work correctly, neither does rebooting.  I have a hunch that
>>something in a cache somewhere may have expired or one of the flags
>>in the /proc tree changed but I sure don't know what.
> 
> 
> The /proc tree is dynamic; it does not persist across reboots. If 
> something is changing it, it's being done by the OS startup scripts.
> 
> 
>>I'm using unpatched iptables-1.2.11 on fc4 with unmodified kernel.
> 
> 
> Very close to 100% chance that it will work when set up properly. A 
> common error is to omit something important from a custom kernel, but 
> distro kernels tend not to have those problems.
> 
> 
>>Box is setup as a router with a WAN, DMZ, and LAN.  WAN traffic
>>DNAT'd to DMZ works. DMZ and LAN through WAN works. The problems show
>>up in LAN to DMZ traffic.
> 
> 
> Incomplete description. Does the LAN and the DMZ have different 
> netblocks? What IP addresses are being accessed by LAN clients? Are you 
> doing the DNAT for those?
> 
> 
>> From LAN to/through DMZ ping (icmp), dns (udp and tcp), and ssh work
>>fine. pop3 and smtp work, but only after a looong wait, much longer
>>than a dns timeout. http works on one DMZ'd server, but on another
>>webserver with 2 IPs will only connect to one of the IPs (the one
>>that the webserver is NOT listening to, but works correctly for WAN
>>traffic).
> 
> 
> /me reaches for the aspirin bottle
> 
> 
>>Sniffing with tcpdump on DMZ for pop3 or smtp traffic shows
>>syn/ack/ack followed by a minutes long wait.  Sniffing for http on
>>DMZ shows correct traffic for D.M.Z.12, but for D.M.Z.11 never shows
>>up on the DMZ interface (11 and 12 are on the same dev).  Switching
>>the order the addresses are added to the interface has no effect.
>>
>>All nics on all machines are brought up with "ifconfig ethX up" and
>>addresses are attached using "ip addr add a.b.c.d/nm dev ethX". 
>>Default routes are created, and on the router
>>/proc/sys/net/ipv4/ip_forward is set to "1".
>>
>>On all machines ifconfig, ip addr show, and route display expected
>>results.
>>
>>The puzzler is that it worked so well for 4 or 5 days.
> 
> 
> More of it is puzzling to me. I think we missed part of the story.
> 
> 
>>Here is the iptables rule set which gets loaded using
>>iptables-restore.
>>
>>##########  VERY BASIC 3-LEGGED FIREWALL/ROUTER  ###########
>>#
>># [eth0] LAN is L.A.N.1   /24 (private)
>># [eth1] WAN is W.A.N.1-5 /29 (dsl to internet)
> 
> 
> Your iptables box is listening on all of these WAN IP's? Is it a /28?
> 
/29

> 
>># [eth2] DMZ is D.M.Z.1   /24 (servers)
> 
> 
> Are these another RFC 1918 netblock? If so why are you obscuring the 
> IP's?
> 
Intent was not to obfuscate.  It was easier for me to keep track of what 
was coming/going to/from where.

> 
>>*nat
>># remember: only NEW connections go through PREROUTING
>>
>>:PREROUTING ACCEPT [0:0]
>>:POSTROUTING ACCEPT [0:0]
>>:OUTPUT ACCEPT [0:0]
>>:GO_1 - [0:0]
>>:GO_2 - [0:0]
>>:GO_3 - [0:0]
>>:GO_4 - [0:0]
>>:GO_5 - [0:0]
>>
>># filtering belongs in filter table...
> 
> 
> Amen, brother, preach on!
> 
> 
>>-A PREROUTING -p icmp -j RETURN
> 
> 
> What does a RETURN do from a builtin chain? I don't understand what 
> you're trying to do with that rule. If it works like ACCEPT, in the nat 
> table that just means no NAT will be done.
> 
Correct.  If it's ICMP I want to skip the NAT rules.
> 
>># divvy ip's into chains; it's faster
>>-A PREROUTING -d W.A.N.1 -j GO_1
>>-A PREROUTING -d W.A.N.2 -j GO_2
>>-A PREROUTING -d W.A.N.3 -j GO_3
>>-A PREROUTING -d W.A.N.4 -j GO_4
>>-A PREROUTING -d W.A.N.5 -j GO_5
> 
> 
> So each WAN IP has its own chain in nat PREROUTING. Not sure why.
> 
To keep the chains short

> 
>># round-robin source ip's make visual log inspection easier for me
> 
> 
> But I don't see any logging anyway, are we missing something here?
Wrong choice of words.  Not logging, footprints for my benefit.

> 
> 
>>-A POSTROUTING -o WAN -j SNAT --to-source W.A.N.1-W.A.N.5
> 
> 
> And I don't understand the benefit in this. I'd use a single IP for the 
> SNAT'ed LAN clients and save the others for other uses.
> 
At some point I'd like to do IP based traffic shaping.  The different 
IPs are easier to spot in the tcpdump output.  Alot of what looks 
obscure is me trying to understand the different aspects of networking.

> 
>># DNAT maps
>>#  eg: -I GO_3 -p tcp --dport 80 -j DNAT --to-destination D.M.Z.100
>>#    would map http://W.A.N.3 to http://D.M.Z.100
> 
> 
> Okay, so it's a DNAT-style DMZ using RFC 1918 addresses for the DMZ? I 
> wouldn't do this. I'd use the "real" IP's in the DMZ. Use rules in 
> filter FORWARD to control access from the outside. Less need for 
> aspirin, and more likely to make it Just Work As Intended.
> 
> You have omitted the actual DNAT mapping rules? This is confusing. 
> Please don't obfuscate your issue beyond the ability to troubleshoot 
> it, if you want help with it.
> 
> 
>>-A GO_1 -j DROP
>>-A GO_2 -j DROP
>>-A GO_3 -j DROP
>>-A GO_4 -j DROP
>>-A GO_5 -j DROP
> 
> 
> But brother, practice what thou preacheth! You said you would not do 
> filtering in the nat table, and yet, you do! Why?
> 
Supposed to be RETURNs but haven't done the filter rules yet.

> You have left important parts out, but these DROP rules could be your 
> culprit.
> 
The nat part works well.  Didn't feel need to include all the mappings.

> 
>>COMMIT
>>#
>>*filter
>>
>>:INPUT   DROP [0:0]
>>:FORWARD DROP [0:0]
>>:OUTPUT ACCEPT [0:0]
>>
>># ok, so i'm an idiot. i like to talk to myself
>>-A INPUT -i lo -j ACCEPT
>># wan is shy
>>-A INPUT -i WAN -p icmp -j DROP
> 
> 
> Not a good idea to put this rule here. Some ICMP you will want to get, 
> even if you block certain types.
> 
Agreed.  Stopgap measure.  Want the basics to work before fleshing 
everything else out.

> 
>># but the rest of us aren't :)
>>-A INPUT -p icmp -j ACCEPT
>># allow router administration from lan
>>-A INPUT -s L.A.N.0/255.255.255.0 -d L.A.N.1 -p tcp -m tcp --dport 22
>>-j ACCEPT
> 
> 
> No --state RELATED,ESTABLISHED -j ACCEPT rule here? I use a "State" 
> chain with this rule, and jump to it from both INPUT and FORWARD, at 
> (or very near) the top of both chains. This would allow you to get the 
> ICMP replies as alluded above.
> 
I don't understand.  How would I get a NEW ssh connection then?

> 
>># let it route...
>>-A FORWARD -o DMZ -j ACCEPT
>># let it route...
>>-A FORWARD -o WAN -j ACCEPT
>># lan offers no services
>>-A FORWARD -o LAN -m state --state RELATED,ESTABLISHED -j ACCEPT
>>COMMIT
> 
> 
> I think if you trace what is happening to each connection you might 
> figure this out. But still, I'd recommend doing away with the DNAT DMZ 
> and using routed IP's instead.

Could you provide one example rule for routed IPs?  That might be enough 
to get me started.


FOLLOW-UP:
OK, I've found what may be causing my symptoms, but don't understand why 
it is happening.

CLIENT <---> LAN 192.168.0.1 <---> DMZ 10.0.0.1 <---> SERVERS

Sniffing with tcpdump at DMZ while telneting from CLIENT:

telnet 10.0.0.12 80 shows client.example.com.27642 > 10.0.0.12.http (OK)
telnet 10.0.0.11 80 shows dmz.example.com.57298 > 10.0.0.11.http (WRONG)

IOW, when telneting to 10.0.0.11.http the router gets the source address 
  wrong, but when telneting to 10.0.0.12.http the source address is correct.

What could cause that to happen?

Thanks again,
San Jose Mike

Re: some addresses won't route

[/dev/rob0]

On Sunday 18 September 2005 17:08, iptables-user wrote:
> /dev/rob0 wrote:
> >>Box is setup as a router with a WAN, DMZ, and LAN.  WAN traffic
> >>DNAT'd to DMZ works. DMZ and LAN through WAN works. The problems
> >> show up in LAN to DMZ traffic.
> >
> > Incomplete description. Does the LAN and the DMZ have different
> > netblocks? What IP addresses are being accessed by LAN clients? Are
> > you doing the DNAT for those?

This be talk-like-a-pirate day, lad, so we be keel-haulin' yer router! 
Arrr! http://www.talklikeapirate.com/

Let's revisit this part, arrrr. DMZ be 10.0.0.0/24 and LAN be 
192.168.0.0/24, aye? Arrr, ye scurvy LAN-lubbers. When a LAN-lubber 
client tries t' hoist the Jolly Roger on server $X it not be goin' 
direct t' IP 10.0.0.$X, be it? It ere goin' to W.A.N.$X, aye?

What ye be doin' to a packet from 192.168.0.101 a-sailin' t' W.A.N.$X? 
It sails t' this router, 192.168.0.1. What ere ye doing with it there? 
Do ye send it out to yer ISP, or does it get DNAT'ed t' 10.0.0.$X? 
That's probably yer answer t' make this scurvy bilge rat work smartly: 
do the DNAT before sendin' it out on the Seven Seas.

> >> From LAN to/through DMZ ping (icmp), dns (udp and tcp), and ssh
> >> work fine. pop3 and smtp work, but only after a looong wait, much
> >> longer than a dns timeout.

This could be happening on the server end, arrr. Perhaps the server is 
checkin' yer reverse DNS, an' those lookups are goin' t' Davy Jones' 
locker.

Are ye runnin' DNS here? That be another way t' make things work 
smartly: ye could have a different internal view in yer nameserver, 
arrr. Ye tell internal clients (LANlubbers) that www.example.com be 
10.0.0.3, and tell external salty dogs that it be W.A.N.3.

I still would recommend the routed configuration, matey. Then ye have 
the same IP's fer all. No quarter! Avast! Arrr!

> >> http works on one DMZ'd server, but on 
> >> another webserver with 2 IPs will only connect to one of the IPs
> >> (the one that the webserver is NOT listening to, but works
> >> correctly for WAN traffic).
> >
> > /me reaches for the aspirin bottle

An' it still hasn't kicked in for that one. Or maybe me grog from last 
night still got the better o' me, arrr!! NAT has t' sail both ways; the 
SNAT'ed connections have t' be DNAT'ed on the journey back t' port. 
Perhaps this not be happenin' with ye multiple IP's.

> >># [eth0] LAN is L.A.N.1   /24 (private)
> >># [eth1] WAN is W.A.N.1-5 /29 (dsl to internet)
> >
> > Your iptables box is listening on all of these WAN IP's? Is it a
> > /28?
>
> /29

Aye, ye said that. Avast!

> >># [eth2] DMZ is D.M.Z.1   /24 (servers)
> >
> > Are these another RFC 1918 netblock? If so why are you obscuring
> > the IP's?
>
> Intent was not to obfuscate.  It was easier for me to keep track of
> what was coming/going to/from where.

Avast! We still not be knowin' what yer flagship be a-doin'!

> >>*nat
> >># remember: only NEW connections go through PREROUTING
> >>
> >>:PREROUTING ACCEPT [0:0]
> >>:POSTROUTING ACCEPT [0:0]
> >>:OUTPUT ACCEPT [0:0]
> >>:GO_1 - [0:0]
> >>:GO_2 - [0:0]
> >>:GO_3 - [0:0]
> >>:GO_4 - [0:0]
> >>:GO_5 - [0:0]
> >>
> >># filtering belongs in filter table...
> >
> > Amen, brother, preach on!
> >
> >>-A PREROUTING -p icmp -j RETURN
> >
> > What does a RETURN do from a builtin chain? I don't understand what
> > you're trying to do with that rule. If it works like ACCEPT, in the
> > nat table that just means no NAT will be done.
>
> Correct.  If it's ICMP I want to skip the NAT rules.

That means ye be not doin' ICMP NAT for ye LANlubbers, arrr. I thought 
ye said 'twas workin' t' ping the DMZ? Was that by IP, lad? Which IP, 
the external ones, or the 10.0.0.0/24 ones? Arrr!

I reckon those be a-workin' smartly.

> >># divvy ip's into chains; it's faster
> >>-A PREROUTING -d W.A.N.1 -j GO_1
> >>-A PREROUTING -d W.A.N.2 -j GO_2
> >>-A PREROUTING -d W.A.N.3 -j GO_3
> >>-A PREROUTING -d W.A.N.4 -j GO_4
> >>-A PREROUTING -d W.A.N.5 -j GO_5
> >
> > So each WAN IP has its own chain in nat PREROUTING. Not sure why.
>
> To keep the chains short

An' more to the point, here be where yer DNAT rules sail, an' ye be 
breakin' them up by IP. That makes sense, but without seein' what ye be 
doin', it be hard t' tell why she's not a'sailin' right proper! Arrrr!

> >>-A GO_5 -j DROP
> >
> > But brother, practice what thou preacheth! You said you would not
> > do filtering in the nat table, and yet, you do! Why?
>
> Supposed to be RETURNs but haven't done the filter rules yet.

RETURN rules be like barnacles on ye hull at the end of a chain. Ye 
don't need 'em there, arrr! Make 'em walk the plank!

> > You have left important parts out, but these DROP rules could be
> > your culprit.
>
> The nat part works well.  Didn't feel need to include all the
> mappings.

Well shiver me timbers, matey, I can't rightly say, arrr!

> >>-A INPUT -i WAN -p icmp -j DROP
> >
> > Not a good idea to put this rule here. Some ICMP you will want to
> > get, even if you block certain types.
>
> Agreed.  Stopgap measure.  Want the basics to work before fleshing
> everything else out.

ACCEPT all yer ICMP as a stopgap measure, me bucko. Aye! It'll help ye 
t' figure things out.

> >># but the rest of us aren't :)
> >>-A INPUT -p icmp -j ACCEPT
> >># allow router administration from lan
> >>-A INPUT -s L.A.N.0/255.255.255.0 -d L.A.N.1 -p tcp -m tcp --dport
> >> 22 -j ACCEPT
> >
> > No --state RELATED,ESTABLISHED -j ACCEPT rule here? I use a "State"
> > chain with this rule, and jump to it from both INPUT and FORWARD,
> > at (or very near) the top of both chains. This would allow you to
> > get the ICMP replies as alluded above.
>
> I don't understand.  How would I get a NEW ssh connection then?

I mean in addition t' yer SSH rule, not a replacement, arrrr. Hoist it 
up yer mainmast alongside the Jolly Roger!

> >># let it route...
> >>-A FORWARD -o DMZ -j ACCEPT
> >># let it route...
> >>-A FORWARD -o WAN -j ACCEPT
> >># lan offers no services
> >>-A FORWARD -o LAN -m state --state RELATED,ESTABLISHED -j ACCEPT
> >>COMMIT
> >
> > I think if you trace what is happening to each connection you might
> > figure this out. But still, I'd recommend doing away with the DNAT
> > DMZ and using routed IP's instead.
>
> Could you provide one example rule for routed IPs?  That might be
> enough to get me started.

I not be talkin' about iptables rules, matey. How be yer /29 delivered 
by the ISP? Do they have a router sittin' on one of yer IP's? If so ye 
can do proxy ARP for the schooners sailin' in the DMZ.

We're getting down into layer 2 territory here, and me own knowledge 
there is a bit short, arrr. Ye don't hoist those IP's on yer flagship 
mainmast, but ye take 'em from the ISP router and pass 'em along to the 
DMZ schooner which be a-listenin' on that IP. Arrrr!

Ye could also just put the DMZ a-sailin' on the same WAN segment, but 
there ye would lose the benefit of controllin' the bilge water through 
yer flagship.

If the ISP be just deliverin' all those IP's directly to ye, ye don't 
need proxy ARP. Regular ARP will work. Yer flagship maintains an ARP 
cache to know which MAC address be answerin' for which IP address, an' 
it all sails right smartly.

> FOLLOW-UP:
> OK, I've found what may be causing my symptoms, but don't understand
> why it is happening.
>
> CLIENT <---> LAN 192.168.0.1 <---> DMZ 10.0.0.1 <---> SERVERS
>
> Sniffing with tcpdump at DMZ while telneting from CLIENT:
>
> telnet 10.0.0.12 80 shows client.example.com.27642 > 10.0.0.12.http
> (OK) telnet 10.0.0.11 80 shows dmz.example.com.57298 > 10.0.0.11.http
> (WRONG)
>
> IOW, when telneting to 10.0.0.11.http the router gets the source
> address wrong, but when telneting to 10.0.0.12.http the source
> address is correct.
>
> What could cause that to happen?

Somethin' be gettin' NAT'ed, matey! Ere ye may be settin' yer source 
address in yer routin' tables. We be not knowin' enough t' guess, arr!
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header

Re: some addresses won't route

[iptables-user]

/dev/rob0 wrote:
> On Sunday 18 September 2005 17:08, iptables-user wrote:
> 
>>/dev/rob0 wrote:
>>
>>>>Box is setup as a router with a WAN, DMZ, and LAN.  WAN traffic
>>>>DNAT'd to DMZ works. DMZ and LAN through WAN works. The problems
>>>>show up in LAN to DMZ traffic.
>>>
>>>Incomplete description. Does the LAN and the DMZ have different
>>>netblocks? What IP addresses are being accessed by LAN clients? Are
>>>you doing the DNAT for those?
> 
> 
> This be talk-like-a-pirate day, lad, so we be keel-haulin' yer router! 
> Arrr! http://www.talklikeapirate.com/
> 
<snip />

Thanks /dev/rob0!!!  I laughed so hard I felt like *I* was full of grog. 
  (I'm part of a small circle of friends who talk like a pirate on a 
regular basis. P)

Sorry for the delay in following up.  My ship dropped me on a jungle 
isle replete with very hungry alligators.  Kept me busy all day :)

Everything now works without delay.  Apparently specifying the output 
interface in a FORWARD chain is not enough.  By modifying the rule to 
include -i AND -o things go where they're supposed to.  Everything now 
works again except LAN traffic to the one DMZ ip:http.

As for the one IP that just won't work for HTTP I think I have a .config 
file somewhere that is conflicting with the network's new identities. 
I've recycled my boxes a couple of times and think that therein lies the 
problem.  It's not that it's being DNAT'd incorrectly, I think it has an 
identity crisis.  The current DMZ IP used to be my client box's IP on a 
different network.  Now that I know it works I can do a clean install, 
copy over the iptables rules, and flesh out the iptables rule set.

And /dev/rob0, thanks again for your detailed help and especially for my 
laugh of the day!!!  (I owe you a good pirate joke)

Here is the working iptables rule set in iptables-save format (WAN IPs 
obfuscated, some comments):

##########  VERY BASIC 3-LEGGED FIREWALL/ROUTER  ###########
#
# [eth0] LAN is 192.168.0.1 /24 (private)
# [eth1] WAN is W.A.N.1-5   /29 (dsl to internet)
# NETWORK:W.A.N.0; IPADDR:W.A.N.1-5; GATEWAY:W.A.N.6; BROADCAST:W.A.N.7
# [eth2] DMZ is 10.0.0.1    /24 (servers)
#
*nat
:PREROUTING ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:WAN_1 - [:]
:WAN_2 - [:]
:WAN_3 - [:]
:WAN_4 - [:]
:WAN_5 - [:]
## break each WAN IP into its own chain
-A PREROUTING -d W.A.N.1 -j WAN_1
-A PREROUTING -d W.A.N.2 -j WAN_2
-A PREROUTING -d W.A.N.3 -j WAN_3
-A PREROUTING -d W.A.N.4 -j WAN_4
-A PREROUTING -d W.A.N.5 -j WAN_5
-A POSTROUTING -o eth1 -j SNAT --to-source W.A.N.5
## dns, pop3, smtp, lot's of different types of webservers
-A WAN_1 -p tcp --dport 25 -j DNAT --to-destination 10.0.0.11
-A WAN_1 -p tcp --dport 53 -j DNAT --to-destination 10.0.0.11
-A WAN_1 -p udp --dport 53 -j DNAT --to-destination 10.0.0.11
-A WAN_1 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.11
-A WAN_2 -p tcp --dport 25 -j DNAT --to-destination 10.0.0.13
-A WAN_2 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.13
-A WAN_2 -p tcp --dport 110 -j DNAT --to-destination 10.0.0.13
-A WAN_3 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.14
-A WAN_4 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.15
-A WAN_5 -p tcp --dport 80 -j DNAT --to-destination 10.0.0.16
COMMIT

*filter
:INPUT   DROP [0:0]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [0:0]

-A INPUT -i lo -j ACCEPT
-A INPUT -p icmp -j ACCEPT
## only LAN hosts can SSH to router
-A INPUT -s 192.168.0.0/255.255.255.0 -d 192.168.0.1 -p tcp -m tcp 
--dport 22 -j ACCEPT

## all devices can go thru DMZ
-A FORWARD -i eth0 -o eth2 -j ACCEPT
-A FORWARD -i eth1 -o eth2 -j ACCEPT
## all devices can go thru WAN
-A FORWARD -i eth0 -o eth1 -j ACCEPT
-A FORWARD -i eth2 -o eth1 -j ACCEPT
## LAN must expose AUTH for access to DMZ's SMTP server...
-A FORWARD -i eth2 -o eth0 -p tcp -m tcp --dport 113 -j ACCEPT
## ...otherwise LAN access is restricted
-A FORWARD -o eth0 -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT

I hope this can be a useful starting point for somebody else.

Thanks to all for your help,
San Jose Mike

Re: some addresses won't route

[/dev/rob0]

On Monday 19 September 2005 19:38, iptables-user wrote:
> Thanks /dev/rob0!!!  I laughed so hard I felt like *I* was full of
> grog. (I'm part of a small circle of friends who talk like a pirate
> on a regular basis. P)

That's a good idea! I get rusty waiting for just one day a year. Brings 
to mind another holiday which should really be more than just one day: 
April Fools' Day.

Alas, me hook an' pegleg an' eyepatch an' cutlass are all stowed away.

> Everything now works without delay.  Apparently specifying the output
> interface in a FORWARD chain is not enough.  By modifying the rule to
> include -i AND -o things go where they're supposed to.  Everything

I wondered about this. I usually just do this by -i interface, and the 
return is covered by the RELATED,ESTABLISHED rule. But I thought your 
way should have worked too.
-- 
    mail to this address is discarded unless "/dev/rob0"
    or "not-spam" is in Subject: header