SO_BINDTODEVICE and IP_TRANSPARENT (TPROXY)

SO_BINDTODEVICE and IP_TRANSPARENT (TPROXY)

[Ashwani Wason]

It appears that SO_BINDTODEVICE is not working when used along with
IP_TRANSPARENT of TPROXY. The use case is for a transparent proxy that
has to handle clients with overlapping IP addresses coming in on
different interfaces (VLANs or physical). Each such interface has a
unique next hop gateway.

Setup: Kernel 2.6.30.9 built with required TPROXY configuration. A
system with three interfaces, eth0 (192.168.0.65/24), eth0.1
(10.0.7.65/16), and eth1 (192.168.1.65/24), each with its own default
router and a fairly standard set of TPROXY rules:

default via 192.168.1.62 dev eth1
default via 192.168.0.62 dev eth0
default via 10.0.0.9 dev eth0.1

iptables -t mangle -N DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
--tproxy-mark 0x1/0x1 --on-port 4002

ip rule add fwmark 1 iif eth0 lookup 101
ip rule add fwmark 1 iif eth1 lookup 101

ip route add local 0.0.0.0/0 dev lo table 101

Additional information:

* The proxy has a separate listening socket for each interface.

* On each listening socket the proxy sets the IP_TRANSPARENT option.
It also sets the SO_BINDTODEVICE option (on each socket) to the
associated interface. (I have experimented with the ordering of these
options and setting one before the other has no change in behavior.)

* The proxy works perfectly fine in "transparent mode" when
SO_BINDTODEVICE is not used.

* The proxy works perfectly fine in "explicit mode" when
SO_BINDTODEVICE *is used*.

Now my test is very simple with the expectation that when a SYN comes
in on, say, interface eth0 then the SYN/ACK must go out back on that
interface to the MAC address of the designated next hop router. That
does not seem to happen however. The SYN/ACK is not being sent out at
all, from any interface. It seems that either the SYN or the SYN/ACK
is being eaten up somewhere. No one 'netstat -s' seems to be updated
either. (Since even 'netstat -ant' does not show the TCP connection
state, I suspect that the SYN is being eaten up.)

When I reconfigure the proxy to be in explicit mode, i.e. not set the
IP_TRANSPARENT option, and have the clients connect directly to the IP
address and port of the proxy then the system works as expected, i.e.
the SYN/ACK (and all subsequent packets) go out on the associated
interface.

Thank you,
Ashwani

Re: SO_BINDTODEVICE and IP_TRANSPARENT (TPROXY)

[Ashwani Wason]

After debugging a bit into how nf_tproxy_get_sock_v4() looks up a
listener for foreign connections using __inet_lookup_listener() and
compute_score() I found that the problem was happening because of the
way I had the TPROXY rules setup. Those rules work if the proxy has a
single listening socket for INADDR_ANY. If multiple listening sockets
must be used, one for each local address, which is the case for using
SO_BINDTODEVICE then the TPROXY rules must also be "fully qualified"
(with interface name [-i] and IP address thereof [--on-ip]). So the
rules in my example change as follows...

Instead of using:

> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 4002

Use:

iptables -t mangle -A PREROUTING *-i eth0* -p tcp --dport 80 -j TPROXY
--tproxy-mark 0x1/0x1 *--on-ip 192.168.0.65* --on-port 4002
iptables -t mangle -A PREROUTING *-i eth1* -p tcp --dport 80 -j TPROXY
--tproxy-mark 0x1/0x1 *--on-ip 192.168.1.65* --on-port 4002

With these rules the listening socket was looked up correctly and the
sk_bound_dev_if of the socket was honored.

- Ashwani

PS: Sorry to anyone (Jamal) who already spent any time on this.
PPS: Copying the tproxy list in case someone else is looking for this
stuff in the future.


On Mon, Nov 30, 2009 at 3:15 PM, Ashwani Wason <ashwas@gmail.com> wrote:
> It appears that SO_BINDTODEVICE is not working when used along with
> IP_TRANSPARENT of TPROXY. The use case is for a transparent proxy that
> has to handle clients with overlapping IP addresses coming in on
> different interfaces (VLANs or physical). Each such interface has a
> unique next hop gateway.
>
> Setup: Kernel 2.6.30.9 built with required TPROXY configuration. A
> system with three interfaces, eth0 (192.168.0.65/24), eth0.1
> (10.0.7.65/16), and eth1 (192.168.1.65/24), each with its own default
> router and a fairly standard set of TPROXY rules:
>
> default via 192.168.1.62 dev eth1
> default via 192.168.0.62 dev eth0
> default via 10.0.0.9 dev eth0.1
>
> iptables -t mangle -N DIVERT
> iptables -t mangle -A DIVERT -j MARK --set-mark 1
> iptables -t mangle -A DIVERT -j ACCEPT
> iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY
> --tproxy-mark 0x1/0x1 --on-port 4002
>
> ip rule add fwmark 1 iif eth0 lookup 101
> ip rule add fwmark 1 iif eth1 lookup 101
>
> ip route add local 0.0.0.0/0 dev lo table 101
>
> Additional information:
>
> * The proxy has a separate listening socket for each interface.
>
> * On each listening socket the proxy sets the IP_TRANSPARENT option.
> It also sets the SO_BINDTODEVICE option (on each socket) to the
> associated interface. (I have experimented with the ordering of these
> options and setting one before the other has no change in behavior.)
>
> * The proxy works perfectly fine in "transparent mode" when
> SO_BINDTODEVICE is not used.
>
> * The proxy works perfectly fine in "explicit mode" when
> SO_BINDTODEVICE *is used*.
>
> Now my test is very simple with the expectation that when a SYN comes
> in on, say, interface eth0 then the SYN/ACK must go out back on that
> interface to the MAC address of the designated next hop router. That
> does not seem to happen however. The SYN/ACK is not being sent out at
> all, from any interface. It seems that either the SYN or the SYN/ACK
> is being eaten up somewhere. No one 'netstat -s' seems to be updated
> either. (Since even 'netstat -ant' does not show the TCP connection
> state, I suspect that the SYN is being eaten up.)
>
> When I reconfigure the proxy to be in explicit mode, i.e. not set the
> IP_TRANSPARENT option, and have the clients connect directly to the IP
> address and port of the proxy then the system works as expected, i.e.
> the SYN/ACK (and all subsequent packets) go out on the associated
> interface.
>
> Thank you,
> Ashwani
>

Re: SO_BINDTODEVICE and IP_TRANSPARENT (TPROXY)

[Eric Dumazet]

Ashwani Wason a écrit :
> After debugging a bit into how nf_tproxy_get_sock_v4() looks up a
> listener for foreign connections using __inet_lookup_listener() and
> compute_score() I found that the problem was happening because of the
> way I had the TPROXY rules setup. Those rules work if the proxy has a
> single listening socket for INADDR_ANY. If multiple listening sockets
> must be used, one for each local address, which is the case for using
> SO_BINDTODEVICE then the TPROXY rules must also be "fully qualified"
> (with interface name [-i] and IP address thereof [--on-ip]). So the
> rules in my example change as follows...
> 
> Instead of using:
> 
>> iptables -t mangle -A PREROUTING -p tcp --dport 80 -j TPROXY --tproxy-mark 0x1/0x1 --on-port 4002
> 
> Use:
> 
> iptables -t mangle -A PREROUTING *-i eth0* -p tcp --dport 80 -j TPROXY
> --tproxy-mark 0x1/0x1 *--on-ip 192.168.0.65* --on-port 4002
> iptables -t mangle -A PREROUTING *-i eth1* -p tcp --dport 80 -j TPROXY
> --tproxy-mark 0x1/0x1 *--on-ip 192.168.1.65* --on-port 4002
> 
> With these rules the listening socket was looked up correctly and the
> sk_bound_dev_if of the socket was honored.
> 
> - Ashwani
> 
> PS: Sorry to anyone (Jamal) who already spent any time on this.
> PPS: Copying the tproxy list in case someone else is looking for this
> stuff in the future.

Thanks a lot for sharing your discoveries :)