[LARTC] multiple uplinks/iptables -t nat -PREROUTING funny

[LARTC] multiple uplinks/iptables -t nat -PREROUTING funny

[Russell Senior]

I have a network much like that show in:

  <http://www.lartc.org/howto/lartc.rpdb.multiple-links.html>

with a few wrinkles.  This is with a ptrace patched 2.4.20 kernel.

The wrinkles are that one interface has 14 IPs (.2 to .15) total (via
eth1 and aliasing), the other being a single DHCP-provided IP (eth0).

I have a iptables rule:

  /sbin/iptables -t nat -A PREROUTING -i eth1 -d eee.fff.ggg.11 -p tcp --dport 22 -j DNAT --to 192.168.0.2:22

That's all there is (currently) in the PREROUTING iptable.
POSTROUTING has:

  /sbin/iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
  /sbin/iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE

When I connect from the outside to eee.fff.ggg.11:22, I never hear a
reply.  Ethereal (on the internal interface, eth2) shows replies, but
looking on the other external interface (eth0) it appears that the
replies are all shooting out of it (the default path).  Nothing ever
returns out the eth1 interface as it "should".  I have the ip rules
set up as follows:

   # ip rule show
   0:      from all lookup local 
   32757:  from aaa.bbb.ccc.ddd lookup T1
   32758:  from eee.fff.ggg.2/31 lookup T2  # note the 3 rules are to cover
   32759:  from eee.fff.ggg.4/30 lookup T2  # the .2 to .15 IP space
   32760:  from eee.fff.ggg.8/29 lookup T2 
   32766:  from all lookup main 
   32767:  from all lookup default 

   # ip route show table T1
   192.168.0.0/24 dev eth2  scope link 
   default via aaa.bbb.ccc.1 dev eth0 

   # ip route show table T2
   eee.fff.ggg.1 dev eth1  scope link 
   eee.fff.ggg.0/28 dev lo  scope link 
   192.168.0.0/24 dev eth2  scope link 
   default via eee.fff.ggg.1 dev eth1 

   x# ip route    # (default rule)
   eee.fff.ggg.1 dev eth1  scope link 
   eee.fff.ggg.0/28 dev lo  scope link 
   192.168.0.0/24 dev eth2  proto kernel  scope link  src 192.168.0.1 
   eee.fff.ggg.0/24 dev eth1  proto kernel  scope link  src eee.fff.ggg.10 
   aaa.bbb.ccc.0/23 dev eth0  proto kernel  scope link  src aaa.bbb.ccc.ddd
   default via aaa.bbb.ccc.1 dev eth0 

When I connect to the port forwarded address from the outside, it
looks like the returning packets are getting routed _before_ the
source IP is translated (and thus aren't matching a special rule and
thus get routed according to the default rule).  Everything else seems
to be working fine. 

Has anyone seen this?  Is it a bug or am I just confused?


-- 
Russell Senior         ``I've seen every kind of critter God ever made,
seniorr@aracnet.com      and I ain't never seen a meaner, lower, more
                         stinkin' yellow hypocrite than you!'' 
                                        -- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] multiple uplinks/iptables -t nat -PREROUTING funny

[Martin A. Brown]

Russell,

 : I have a network much like that show in:
 :
 :   <http://www.lartc.org/howto/lartc.rpdb.multiple-links.html>
 :
 : with a few wrinkles.  This is with a ptrace patched 2.4.20 kernel.
 :
 : The wrinkles are that one interface has 14 IPs (.2 to .15) total (via
 : eth1 and aliasing), the other being a single DHCP-provided IP (eth0).

As a side note, after reading your post, Russell, I'm a touch confused why
you have so many IPs hosted on your eth1 interface.  It strikes me that
you really just want packets to arrive (inbound) on eth1.  So, you really
don't need to have all these IPs locally hosted on your box.  You can
simply proxy ARP for the IPs, and ethernet frames with IP packets bound
for e.f.g.2-15 will arrive on your eth1.  Perhaps you are doing more than
that with these 14 IP addresses, but if not, why bother hosting the IPs
locally?

Suggestion:

# for h in $( seq 2 15 ) ; do
>   arp -s e.f.g.$h -i eth1 -D eth1 pub
> done

Purely out of curiousity, I wonder if you are doing something else with
these IPs?

[ snipped iptables rules that look fine ]

 : When I connect from the outside to eee.fff.ggg.11:22, I never hear a
 : reply.  Ethereal (on the internal interface, eth2) shows replies, but
 : looking on the other external interface (eth0) it appears that the
 : replies are all shooting out of it (the default path).  Nothing ever
 : returns out the eth1 interface as it "should".  I have the ip rules
 : set up as follows:

[ thank you for the "ip route" and "ip rule" output, which has been
  snipped to save space  ]

 : When I connect to the port forwarded address from the outside, it
 : looks like the returning packets are getting routed _before_ the
 : source IP is translated (and thus aren't matching a special rule and
 : thus get routed according to the default rule).  Everything else seems
 : to be working fine.
 :
 : Has anyone seen this?  Is it a bug or am I just confused?

This is not a bug--this is a fact of packet flow through the kernel.  See
the kernel packet traveling diagram (KPTD) [1] for more details on the
sequence of operations.  So to answer your question:  you must be
confused!  :)

You should try adding just one more rule:

# ip rule add from 192.168.0.2 table T2

For a more thorough analysis of this interaction between routing and
netfilter, see my advanced routing with multiple Internet connections
chapter [2] and/or this informative post from Wes Hodges [3].

-Martin

 [1]  http://www.docum.org/stef.coene/qos/kptd/
 [2]  http://linux-ip.net/html/adv-multi-internet.html
 [3]  http://lists.netfilter.org/pipermail/netfilter/2001-May/011697.html

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] multiple uplinks/iptables -t nat -PREROUTING funny

[Russell Senior]

>>>>> "Martin" = Martin A Brown <mabrown-lartc@securepipe.com> writes:

Russell> When I connect to the port forwarded address from the
Russell> outside, it looks like the returning packets are getting
Russell> routed _before_ the source IP is translated (and thus aren't
Russell> matching a special rule and thus get routed according to the
Russell> default rule).  Everything else seems to be working fine.

Russell> Has anyone seen this?  Is it a bug or am I just confused?

Martin> This is not a bug--this is a fact of packet flow through the
Martin> kernel.  See the kernel packet traveling diagram (KPTD) [1]
Martin> for more details on the sequence of operations.  So to answer
Martin> your question: you must be confused!  :)

What that very nice diagram doesn't show is how the reply packets to
DNAT'd connections are handled.  The prima facie evidence seems to be
that DNAT was in the PREROUTING iptable and "consequently" the reverse
translation should occur before routing.  That is the source of my
confusion.

Martin> You should try adding just one more rule:

Martin> # ip rule add from 192.168.0.2 table T2

That would "work", but it is kind of messy.  What if I have a second
DNAT from IF1 that also forwards to 192.168.0.2?  It would get
complicated in a hurry.  

All would be solved if the reverse translation just occurred in
PREROUTING as seems like it "should".  I don't understand yet why it
doesn't.  Perhaps there is a good reason that I just don't see.  Or,
maybe there isn't a good reason, and it should be "fixed".  Too soon
for me to say.

If anyone can point me at some detailed documentation on DNAT or even
the relevant bits of the source code, I'd really appreciate it!

-- 
Russell Senior         ``I've seen every kind of critter God ever made,
seniorr@aracnet.com      and I ain't never seen a meaner, lower, more
                         stinkin' yellow hypocrite than you!'' 
                                        -- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] multiple uplinks/iptables -t nat -PREROUTING funny

[Russell Senior]

>>>>> "Russell" = Russell Senior <seniorr@aracnet.com> writes:

>>>>> "Martin" = Martin A Brown <mabrown-lartc@securepipe.com> writes:

Russell> When I connect to the port forwarded address from the
Russell> outside, it looks like the returning packets are getting
Russell> routed _before_ the source IP is translated (and thus aren't
Russell> matching a special rule and thus get routed according to the
Russell> default rule).  Everything else seems to be working fine.

Russell> Has anyone seen this?  Is it a bug or am I just confused?

Martin> This is not a bug--this is a fact of packet flow through the
Martin> kernel.  See the kernel packet traveling diagram (KPTD) [1]
Martin> for more details on the sequence of operations.  So to answer
Martin> your question: you must be confused!  :)

Russell> What that very nice diagram doesn't show is how the reply
Russell> packets to DNAT'd connections are handled.  The prima facie
Russell> evidence seems to be that DNAT was in the PREROUTING iptable
Russell> and "consequently" the reverse translation should occur
Russell> before routing.  That is the source of my confusion.

Martin> You should try adding just one more rule:

Martin> # ip rule add from 192.168.0.2 table T2

Russell> That would "work", but it is kind of messy.  What if I have a
Russell> second DNAT from IF1 that also forwards to 192.168.0.2?  It
Russell> would get complicated in a hurry.

Russell> All would be solved if the reverse translation just occurred
Russell> in PREROUTING as seems like it "should".  I don't understand
Russell> yet why it doesn't.  Perhaps there is a good reason that I
Russell> just don't see.  Or, maybe there isn't a good reason, and it
Russell> should be "fixed".  Too soon for me to say.

Just as a followup to this: a relatively clean solution is to mark the
reply packets:

  iptables -t mangle -I PREROUTING -m conntrack --ctstate DNAT --ctorigdst eee.fff.ggg.11 -j MARK --set-mark 2

  ip rule add fwmark 2 table T2

and make sure that rp_filter is appropriately off:

  echo 0 > /proc/sys/net/ipv4/conf/<if>/rp_filter



-- 
Russell Senior         ``I've seen every kind of critter God ever made,
seniorr@aracnet.com      and I ain't never seen a meaner, lower, more
                         stinkin' yellow hypocrite than you!'' 
                                        -- Burl Ives as Rufus Hennessy
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/