How to use TROXY target only for specific outgoing interface

How to use TROXY target only for specific outgoing interface

[Sebastian Poehn]

I want to run a tcp transparent proxy ( with TPROXY ) processing only traffic outgoing a specific interface. That's what my setup looks like:
                                                      
                                                        
                   lan1 ############                                
    local net 1 <-------#  ROUTER  # wan                            
                        #    +     #-------------> internet                       
    local net 2 <-------#  TPROXY  #                            
                   lan2 ############                                
                                                        
Local traffic to lan1 and lan2 shall not be proxied, traffic to wan shall be proxied.
Other routers may be connected to the local networks, so we can not match for any destination netmasks.

The TPROXY target requires being entered in PREROUTING chain. Naturally the outgoing interface is only known after routing (POSTROUTING),
but then it's to late for TPROXY.

I was already thinking of possible solutions but all look more or less quirky to me (just pseudo calls, please do not insist on correctness ;-)

Solution 1 (run the stack twice):
ip link add dev loopback name tprox
iptables -A POSTROUTING -o wan -j ROUTE -oif tprox
iptables -A PREROUTING -i tprox -j TPROXY

Solution 2 (do it on your own):
iptables -A PREROUTING -j NFQUEUE
Use a small homebrew program using destination address and performing route lookup. If outgoing interface is wan mark packet and NF_REPEAT.
iptables -A PREROUTING --match-mark X -j PROXY

Solution 3 (just another idea):
TPROXY requires a ip route add local 0.0.0.0/0 (deliver everything locally).
If you'd jump to a the regular routing table traffic will not be proxied.
Unfortunately policy routing (ip rule *) can not 'match' for an outgoing interface (because the route lookup has not happened yet).

It would be great if you could give me any hints.

Greetings
Sebastian Poehn
---

Re: How to use TROXY target only for specific outgoing interface

[Jan Engelhardt]

On Sunday 2013-01-13 09:54, Sebastian Poehn wrote:

>I want to run a tcp transparent proxy ( with TPROXY ) processing only traffic outgoing a specific interface. That's what my setup looks like:
>                                                      
>                                                        
>                   lan1 ############                                
>    local net 1 <-------#  ROUTER  # wan                            
>                        #    +     #-------------> internet                       
>    local net 2 <-------#  TPROXY  #                            
>                   lan2 ############                                
>                                                        

-A PREROUTING -j foo
forall LAN subnets
	-A foo -d $lan -j RETURN
-A foo -j TPROXY

Re: How to use TROXY target only for specific outgoing interface

[Sebastian Poehn]

For a simple setup this is more than sufficient. But I want to realize
something with dynamic routing. So to clarify:

        ospf            lan1 ############                                 
local3 <----> local1 <-------#  ROUTER  # wan                            
                             #    +     #-------------> internet                       
              local2 <-------#  TPROXY  #                            
                        lan2 ############       

For me it's not possible to even know every subnet which is on the local
side. It would even be possible that there is a multi-homed environment
with e.g. local3 connected to the internet, too. (Thank means that even
a non-local destination could go from local2, via lan2, lan1, local1 and
local3 to the "internet" ).

Thank for your reply Jan

On Sun, 2013-01-13 at 12:30 +0100, Jan Engelhardt wrote:
> On Sunday 2013-01-13 09:54, Sebastian Poehn wrote:
> 
> >I want to run a tcp transparent proxy ( with TPROXY ) processing only traffic outgoing a specific interface. That's what my setup looks like:
> >                                                      
> >                                                        
> >                   lan1 ############                                
> >    local net 1 <-------#  ROUTER  # wan                            
> >                        #    +     #-------------> internet                       
> >    local net 2 <-------#  TPROXY  #                            
> >                   lan2 ############                                
> >                                                        
> 
> -A PREROUTING -j foo
> forall LAN subnets
> 	-A foo -d $lan -j RETURN
> -A foo -j TPROXY

Re: How to use TROXY target only for specific outgoing interface

[Eliezer Croitoru]

If you would give an ip example rather then a sketch I think I have an 
idea on how to do it using some local routing daemon on the router machine.

Another thing to notice is that if you are using tproxy it should be 
used based on a known network data or globally with specific exceptions.
else then these situation you will need to plan some iptables structure 
to fit maybe ipset or any other way of organizing the dynamic tproxy rules.

Eliezer

On 1/13/2013 6:39 PM, Sebastian Poehn wrote:
> For a simple setup this is more than sufficient. But I want to realize
> something with dynamic routing. So to clarify:
>
>          ospf            lan1 ############
> local3 <----> local1 <-------#  ROUTER  # wan
>                               #    +     #-------------> internet
>                local2 <-------#  TPROXY  #
>                          lan2 ############
>
> For me it's not possible to even know every subnet which is on the local
> side. It would even be possible that there is a multi-homed environment
> with e.g. local3 connected to the internet, too. (Thank means that even
> a non-local destination could go from local2, via lan2, lan1, local1 and
> local3 to the "internet" ).
>
> Thank for your reply Jan

Re: How to use TROXY target only for specific outgoing interface

[Sebastian Poehn]

I've drawn a new picture. We want to connect from Node 1 to the 'internet'. All traffic via wan1 shall be proxied, traffic over lan1 not.

The only valid match for this situation is the outgoing interface ( oif == wan1 do proxy, else no proxy). It is not possible to match for
dst networks, as routing metrics may change and so even the use of wan1 or wan2 (for the uplink).

I can not -A POSTROUTING -o wan1 -j TPROXY as TPROXY must be called in PREROUTING (there -o is not present).

               internet
  ____________________________________
   A                              A
   |wan1                          |wan2
   |                              |
#########      #########      #########
#ROUTER1# lan1 #ROUTER2# lan3 #ROUTER3#
#   +   #<---->#       #<---->#       #
#TPROXY #  igp #       # igp  #       #
#########      #########      #########
   |
   | lan2
   |
#########
# NODE 1#
#       #
#       #
#########






On Mon, 2013-01-14 at 00:33 +0200, Eliezer Croitoru wrote:
> If you would give an ip example rather then a sketch I think I have an 
> idea on how to do it using some local routing daemon on the router machine.
> 
> Another thing to notice is that if you are using tproxy it should be 
> used based on a known network data or globally with specific exceptions.
> else then these situation you will need to plan some iptables structure 
> to fit maybe ipset or any other way of organizing the dynamic tproxy rules.
> 
> Eliezer
> 
> On 1/13/2013 6:39 PM, Sebastian Poehn wrote:
> > For a simple setup this is more than sufficient. But I want to realize
> > something with dynamic routing. So to clarify:
> >
> >          ospf            lan1 ############
> > local3 <----> local1 <-------#  ROUTER  # wan
> >                               #    +     #-------------> internet
> >                local2 <-------#  TPROXY  #
> >                          lan2 ############
> >
> > For me it's not possible to even know every subnet which is on the local
> > side. It would even be possible that there is a multi-homed environment
> > with e.g. local3 connected to the internet, too. (Thank means that even
> > a non-local destination could go from local2, via lan2, lan1, local1 and
> > local3 to the "internet" ).
> >
> > Thank for your reply Jan

Re: How to use TROXY target only for specific outgoing interface

[Eliezer Croitoru]

On 1/14/2013 9:12 PM, Sebastian Poehn wrote:
> I've drawn a new picture. We want to connect from Node 1 to the 'internet'. All traffic via wan1 shall be proxied, traffic over lan1 not.
>
> The only valid match for this situation is the outgoing interface ( oif == wan1 do proxy, else no proxy). It is not possible to match for
> dst networks, as routing metrics may change and so even the use of wan1 or wan2 (for the uplink).
>
> I can not -A POSTROUTING -o wan1 -j TPROXY as TPROXY must be called in PREROUTING (there -o is not present).
>
>                 internet
>    ____________________________________
>     A                              A
>     |wan1                          |wan2
>     |                              |
> #########      #########      #########
> #ROUTER1# lan1 #ROUTER2# lan3 #ROUTER3#
> #   +   #<---->#       #<---->#       #
> #TPROXY #  igp #       # igp  #       #
> #########      #########      #########
>     |
>     | lan2
>     |
> #########
> # NODE 1#
> #       #
> #       #
> #########
>
>
Hey there,

Thanks for the new picture.
I understand what is your problem since it's a very common concept.
But you must understand that most of TPROXY systems are very static.

Take a deep breath and accept(like a server\socket) that the connections 
can be intercepted only in a prerouting table.
what I do suggest you is to use a synamic ipset to allow you this 
specific thing you need.

Since you can only use ip addresses as a match in the prerouting or an 
incoming interface.
Your problem is that you are using IGP which limits you to a specific 
protocol which I think BIRD or QUAGGA doesn't work with.

What you can do if it was another protocol is to run a cron task every 
once in a while to make sure the routing tables are still the same or 
not and in a case of change to update an ipset that you will use to 
either bypass or intercept the traffic into.

If you are working with ROUTING protocols it should be simple to know 
what traffic is being routed to WAN1.

Best regards,
Eliezer

Re: How to use TROXY target only for specific outgoing interface

[Sebastian Poehn]

Thanks Eliezer. You are right. Creating a ipset containing all routes is
the only thing you can do in PREROUTING. As this solution is not useable
for me, I ended up writing a small piece of code taking a packet from
NFQUEUE and performing a nexthop lookup and outgoing if and accordingly
setting a fwmark.

-m mark --mark LOCAL -j CONNMARK --set-mark LOCAL
-m mark --mark WIDE -j CONNMARK --set-mark WIDE
-m connmark --mark LOCAL -j ACCEPT
-m connmark --mark WIDE -j TPROXY
-m TPROXYTRAFFIC -j NFQUEUE

Notice the usage of connmark so only one lookup is needed for a stream. 

On Tue, 2013-01-15 at 14:02 +0200, Eliezer Croitoru wrote:
> On 1/14/2013 9:12 PM, Sebastian Poehn wrote:
> > I've drawn a new picture. We want to connect from Node 1 to the 'internet'. All traffic via wan1 shall be proxied, traffic over lan1 not.
> >
> > The only valid match for this situation is the outgoing interface ( oif == wan1 do proxy, else no proxy). It is not possible to match for
> > dst networks, as routing metrics may change and so even the use of wan1 or wan2 (for the uplink).
> >
> > I can not -A POSTROUTING -o wan1 -j TPROXY as TPROXY must be called in PREROUTING (there -o is not present).
> >
> >                 internet
> >    ____________________________________
> >     A                              A
> >     |wan1                          |wan2
> >     |                              |
> > #########      #########      #########
> > #ROUTER1# lan1 #ROUTER2# lan3 #ROUTER3#
> > #   +   #<---->#       #<---->#       #
> > #TPROXY #  igp #       # igp  #       #
> > #########      #########      #########
> >     |
> >     | lan2
> >     |
> > #########
> > # NODE 1#
> > #       #
> > #       #
> > #########
> >
> >
> Hey there,
> 
> Thanks for the new picture.
> I understand what is your problem since it's a very common concept.
> But you must understand that most of TPROXY systems are very static.
> 
> Take a deep breath and accept(like a server\socket) that the connections 
> can be intercepted only in a prerouting table.
> what I do suggest you is to use a synamic ipset to allow you this 
> specific thing you need.
> 
> Since you can only use ip addresses as a match in the prerouting or an 
> incoming interface.
> Your problem is that you are using IGP which limits you to a specific 
> protocol which I think BIRD or QUAGGA doesn't work with.
> 
> What you can do if it was another protocol is to run a cron task every 
> once in a while to make sure the routing tables are still the same or 
> not and in a case of change to update an ipset that you will use to 
> either bypass or intercept the traffic into.
> 
> If you are working with ROUTING protocols it should be simple to know 
> what traffic is being routed to WAN1.
> 
> Best regards,
> Eliezer
> 

Re: How to use TROXY target only for specific outgoing interface

[Eliezer Croitoru]

Very nice!!

Can you share this code by any chance?

Eliezer
On 1/15/2013 8:37 PM, Sebastian Poehn wrote:
> Thanks Eliezer. You are right. Creating a ipset containing all routes is
> the only thing you can do in PREROUTING. As this solution is not useable
> for me, I ended up writing a small piece of code taking a packet from
> NFQUEUE and performing a nexthop lookup and outgoing if and accordingly
> setting a fwmark.
>
> -m mark --mark LOCAL -j CONNMARK --set-mark LOCAL
> -m mark --mark WIDE -j CONNMARK --set-mark WIDE
> -m connmark --mark LOCAL -j ACCEPT
> -m connmark --mark WIDE -j TPROXY
> -m TPROXYTRAFFIC -j NFQUEUE
>
> Notice the usage of connmark so only one lookup is needed for a stream.