Using MARK and TOS to route traffic through different interfaces to the same destination

Using MARK and TOS to route traffic through different interfaces to the same destination

[Javier Gálvez Guerrero]

Hi all,

I need to route packets through different interfaces (let them be ath0
and eth0) depending on the application source port, so I thought using
TOS or MARK targets of iptables would be helpful.

Anyway, as I try configure it to mark the traffic and updating the
routing tables through many different ways, I can't get it working so
the packets are always sent through the "default" interface in the
main routing table.

For example, if I use MARK I configure it this way:

sudo iptables -A OUTPUT -t mangle -p tcp --dport 60301 -j MARK --set-mark 1
sudo iptables -A OUTPUT -t mangle -p tcp --dport 60302 -j MARK --set-mark 2

sudo ip rule add fwmark 1 table 1 prio 1
sudo ip rule add fwmark 2 table 2 prio 2

sudo ip route add table 1 nexthop via 192.168.0.1 dev ath0
sudo ip route add table 2 nexthop via 192.168.0.1 dev eth0


The routing tables and the iptables rules are properly updated but
packets I sent with these source ports are always sent through the
default interface in the main routing table (if I change this default
entry then the packets are sent through this again).


Any idea about what I am missing? Any help would be much appreciated.


Thank you,
Javi

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Thomas Jacob]

On Thu, 2008-12-11 at 13:18 +0100, Javier Gálvez Guerrero wrote:
> Hi all,
> 
> I need to route packets through different interfaces (let them be ath0
> and eth0) depending on the application source port, so I thought using
> TOS or MARK targets of iptables would be helpful.
> 
> Anyway, as I try configure it to mark the traffic and updating the
> routing tables through many different ways, I can't get it working so
> the packets are always sent through the "default" interface in the
> main routing table.
> 
> For example, if I use MARK I configure it this way:
> 
> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60301 -j MARK --set-mark 1
> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60302 -j MARK --set-mark 2

AFAIK, locally generated packets are routed before they are sent to
netfilter, so setting fwmarks there to influence routing is pointless.

See http://ebtables.sourceforge.net/br_fw_ia/br_fw_ia.html

Figure 3a

If you can use two different source IPs, you could probably
bind them to each interface and then you wouldn't need policy
routing at all to achieve your objective.


> sudo ip rule add fwmark 1 table 1 prio 1
> sudo ip rule add fwmark 2 table 2 prio 2
> 
> sudo ip route add table 1 nexthop via 192.168.0.1 dev ath0
> sudo ip route add table 2 nexthop via 192.168.0.1 dev eth0
> 
> 

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Pascal Hambourg]

Hello,

Thomas Jacob a écrit :
> On Thu, 2008-12-11 at 13:18 +0100, Javier Gálvez Guerrero wrote:
>>
>> I need to route packets through different interfaces (let them be ath0
>> and eth0) depending on the application source port, so I thought using
>> TOS or MARK targets of iptables would be helpful.
>>
>> Anyway, as I try configure it to mark the traffic and updating the
>> routing tables through many different ways, I can't get it working so
>> the packets are always sent through the "default" interface in the
>> main routing table.
>>
>> For example, if I use MARK I configure it this way:
>>
>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60301 -j MARK --set-mark 1
>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60302 -j MARK --set-mark 2

These rules match the destination port. Replace --dport with --sport to 
match the source port.

> AFAIK, locally generated packets are routed before they are sent to
> netfilter, so setting fwmarks there to influence routing is pointless.

A rerouting happens after the OUTPUT chains in order to take into 
account destination NAT and marks.

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Thomas Jacob]

On Thu, 2008-12-11 at 13:41 +0100, Pascal Hambourg wrote:
> > AFAIK, locally generated packets are routed before they are sent to
> > netfilter, so setting fwmarks there to influence routing is pointless.
> 
> A rerouting happens after the OUTPUT chains in order to take into 
> account destination NAT and marks.

Didn't now that, does this always happen (so all locally generated
packets are routed twice, when iptables is active) or only
when netfilter changes things that might affect the destination
of a packet?

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Philip Craig]

Thomas Jacob wrote:
> Didn't now that, does this always happen (so all locally generated
> packets are routed twice, when iptables is active) or only
> when netfilter changes things that might affect the destination
> of a packet?

Only when netfilter changes things.

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Javier Gálvez Guerrero]

Hi,

2008/12/11 Pascal Hambourg <pascal.mail@plouf.fr.eu.org>:
> Hello,
>
> Thomas Jacob a écrit :
>>
>> On Thu, 2008-12-11 at 13:18 +0100, Javier Gálvez Guerrero wrote:
>>>
>>> I need to route packets through different interfaces (let them be ath0
>>> and eth0) depending on the application source port, so I thought using
>>> TOS or MARK targets of iptables would be helpful.
>>>
>>> Anyway, as I try configure it to mark the traffic and updating the
>>> routing tables through many different ways, I can't get it working so
>>> the packets are always sent through the "default" interface in the
>>> main routing table.
>>>
>>> For example, if I use MARK I configure it this way:
>>>
>>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60301 -j MARK --set-mark
>>> 1
>>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60302 -j MARK --set-mark
>>> 2
>
> These rules match the destination port. Replace --dport with --sport to
> match the source port.

Sorry, 60301 and 60302 are both destination port. I made a mistake
when explaining it. I need to route packets depending on the
DESTINATION port. The two interfaces (eth0 and ath0) are binded to
different IP addresses both in the same range of the router and the
destination host (192.168.0.0/24).

Then, any idea?

Thank you for your help,
Javi

>
>> AFAIK, locally generated packets are routed before they are sent to
>> netfilter, so setting fwmarks there to influence routing is pointless.
>
> A rerouting happens after the OUTPUT chains in order to take into account
> destination NAT and marks.
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Pascal Hambourg]

Javier Gálvez Guerrero a écrit :
>>
>>>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60301 -j MARK --set-mark 1
>>>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60302 -j MARK --set-mark 2
>>
>> These rules match the destination port. Replace --dport with --sport to
>> match the source port.
> 
> Sorry, 60301 and 60302 are both destination port. I made a mistake
> when explaining it. I need to route packets depending on the
> DESTINATION port.

Did you check that the iptables rules actually match packets ? Are the 
associated counters shown by iptables -vL or iptables-save -c incrementing ?

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Javier Gálvez Guerrero]

Hi,

It seems that it matches and manages more or less properly, but the IP
address not change while the output interface seems to be selected
accordingly the rules. As said before, each interface is binded to a
different IP address.

This is the configuration I set:

sudo ifconfig ath0 192.168.0.150 netmask 255.255.255.128
sudo ifconfig eth0 192.168.0.3 netmask 255.255.255.128

sudo ip route flush table 1
sudo ip route flush table 2
sudo iptables -F OUTPUT -t mangle

# Mark traffic from port 60301 with 1 and from port 60302 with 2
sudo iptables -A OUTPUT -t mangle -p tcp --dport 60301 -j TOS --set-tos 0x10
sudo iptables -A OUTPUT -t mangle -p tcp --dport 60302 -j TOS --set-tos 0x08

sudo ip rule add tos 0x10 table 1 prio 1
sudo ip rule add tos 0x08 table 2 prio 2

sudo ip route add table 1 192.168.0.2 dev ath0
sudo ip route add table 2 192.168.0.2 dev eth0


This is how the configuration is set:

dulceangustia@spike:~$ sudo iptables --list -t mangle
Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination

Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination
TOS        tcp  --  anywhere             anywhere            tcp
dpt:60301 TOS set Minimize-Delay
TOS        tcp  --  anywhere             anywhere            tcp
dpt:60302 TOS set Maximize-Throughput

Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination
dulceangustia@spike:~$ ip route
192.168.0.0/25 dev eth0  proto kernel  scope link  src 192.168.0.3
192.168.0.128/25 dev ath0  proto kernel  scope link  src 192.168.0.150
default via 192.168.0.1 dev eth0
dulceangustia@spike:~$ sudo ip route show table 1
192.168.0.2 dev ath0  scope link
dulceangustia@spike:~$ sudo ip route show table 2
192.168.0.2 dev eth0  scope link
dulceangustia@spike:~$ sudo ip rule show
0:      from all lookup local
1:      from all tos lowdelay lookup 1
2:      from all tos throughput lookup 2
32766:  from all lookup main
32767:  from all lookup default


And this is what I get:

dulceangustia@spike:~$ sudo iptables -vL
Chain INPUT (policy ACCEPT 80932 packets, 60M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 100K packets, 116M bytes)
 pkts bytes target     prot opt in     out     source
destination
dulceangustia@spike:~$ sudo iperf -c 192.168.0.2 -t 1 -p 60301 -r
------------------------------------------------------------
Server listening on TCP port 60301
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 60301
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.0.3 port 40316 connected with 192.168.0.2 port 60301
[  5]  0.0- 1.0 sec  3.52 MBytes  29.4 Mbits/sec
[  4] local 192.168.0.3 port 60301 connected with 192.168.0.2 port 60077
[  4]  0.0- 1.1 sec  12.1 MBytes  93.8 Mbits/sec
dulceangustia@spike:~$ sudo iptables -vL
Chain INPUT (policy ACCEPT 91086 packets, 73M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 107K packets, 120M bytes)
 pkts bytes target     prot opt in     out     source
destination
dulceangustia@spike:~$ sudo iperf -c 192.168.0.2 -t 1 -p 60302 -r
------------------------------------------------------------
Server listening on TCP port 60302
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 60302
TCP window size: 16.0 KByte (default)
------------------------------------------------------------
[  5] local 192.168.0.3 port 54737 connected with 192.168.0.2 port 60302
[  5]  0.0- 1.0 sec  11.6 MBytes  97.5 Mbits/sec
[  4] local 192.168.0.3 port 60302 connected with 192.168.0.2 port 42854
[  4]  0.0- 1.1 sec  12.6 MBytes  93.8 Mbits/sec
dulceangustia@spike:~$ sudo iptables -vL
Chain INPUT (policy ACCEPT 105K packets, 87M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 121K packets, 133M bytes)
 pkts bytes target     prot opt in     out     source               destination


As you can note, the IP address remains unchanged even the packets
being sent through the correct interface. So the server, according to
its ARP table sends back the packet stream to the interface binded to
the source IP address, this not being the same interface where the
packets came from.

I don't know why the source IP address is not changed. May be a
problem of ip route and not iptables? Any idea about how to solve it?
I tried also to add a POSTROUTING SNAT rule to change the origin
source but it doesn't take any effect.

It's a weird behavior (or I'm missing an important issue in this
packet management).


Regards,
Javi


2008/12/12 Pascal Hambourg <pascal.mail@plouf.fr.eu.org>
>
> Javier Gálvez Guerrero a écrit :
>>>
>>>>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60301 -j MARK --set-mark 1
>>>>> sudo iptables -A OUTPUT -t mangle -p tcp --dport 60302 -j MARK --set-mark 2
>>>
>>> These rules match the destination port. Replace --dport with --sport to
>>> match the source port.
>>
>> Sorry, 60301 and 60302 are both destination port. I made a mistake
>> when explaining it. I need to route packets depending on the
>> DESTINATION port.
>
> Did you check that the iptables rules actually match packets ? Are the associated counters shown by iptables -vL or iptables-save -c incrementing ?
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Pascal Hambourg]

Javier Gálvez Guerrero a écrit :
> 
> It seems that it matches and manages more or less properly, but the IP
> address not change while the output interface seems to be selected
> accordingly the rules. As said before, each interface is binded to a
> different IP address.

The source address is selected either by the sender process or by the 
initial routing decision, before the OUTPUT chains. Rerouting after the 
OUTPUT chains does not alter it even though the output interface has 
changed. Thus the source address selection is unaware of iptables-based 
advanced routing.

> This is the configuration I set:
[...]
> sudo ip route add table 1 192.168.0.2 dev ath0
> sudo ip route add table 2 192.168.0.2 dev eth0

Isn't there a typo ? These commands create host routes to 192.168.0.2, 
not default routes via gateway 192.168.0.2 as in your previous message.
Another problem is that according to the interface subnets and the main 
routing table, 192.168.0.2 is reachable only on eth0, not ath0.

> dulceangustia@spike:~$ ip route
> 192.168.0.0/25 dev eth0  proto kernel  scope link  src 192.168.0.3
> 192.168.0.128/25 dev ath0  proto kernel  scope link  src 192.168.0.150
> default via 192.168.0.1 dev eth0

Are both interfaces on the same link ? If yes, what is the subnet on 
that link ?

> And this is what I get:
> 
> dulceangustia@spike:~$ sudo iptables -vL

This command only shows the default (filter) table. You want to display 
the mangle table with -t mangle.

> I don't know why the source IP address is not changed. May be a
> problem of ip route and not iptables?

I explained why the source address is unchanged, see above.

> Any idea about how to solve it?

Either select the source address in the sender process (if you can 
select the destination port, you may be able to select the source 
address too) or use iptables SNAT.

> I tried also to add a POSTROUTING SNAT rule to change the origin
> source but it doesn't take any effect.

It should work. What rules did you try ? Note that iperf shows only the 
initial source address before SNAT, not the actual address on the wire.

Re: Using MARK and TOS to route traffic through different interfaces to the same destination

[Javier Gálvez Guerrero]

Hi,

Thanks for your answer.

2008/12/12 Pascal Hambourg <pascal.mail@plouf.fr.eu.org>:
> Javier Gálvez Guerrero a écrit :
>>
>> It seems that it matches and manages more or less properly, but the IP
>> address not change while the output interface seems to be selected
>> accordingly the rules. As said before, each interface is binded to a
>> different IP address.
>
> The source address is selected either by the sender process or by the
> initial routing decision, before the OUTPUT chains. Rerouting after the
> OUTPUT chains does not alter it even though the output interface has
> changed. Thus the source address selection is unaware of iptables-based
> advanced routing.

Ok. So, as you said, I must use SNAT or tell the application the
source IP to be used.

>
>> This is the configuration I set:
>
> [...]
>>
>> sudo ip route add table 1 192.168.0.2 dev ath0
>> sudo ip route add table 2 192.168.0.2 dev eth0
>
> Isn't there a typo ? These commands create host routes to 192.168.0.2, not
> default routes via gateway 192.168.0.2 as in your previous message.

In the previous message I used another script with different entries;
it told the host where was the gateway (192.168.0.1. In the later,
what I say is which interface must be used when the packets are to be
sent to host 192.168.0.2 (the server).

> Another problem is that according to the interface subnets and the main
> routing table, 192.168.0.2 is reachable only on eth0, not ath0.

May the problems be related to this issue? By the way, I have set
/proc/sys/net/ipv4/route/min_delay and max_delay values to 0, so
routing changes (should) take effect inmediately, then flushing the
routing cache.

>
>> dulceangustia@spike:~$ ip route
>> 192.168.0.0/25 dev eth0  proto kernel  scope link  src 192.168.0.3
>> 192.168.0.128/25 dev ath0  proto kernel  scope link  src 192.168.0.150
>> default via 192.168.0.1 dev eth0
>
> Are both interfaces on the same link ? If yes, what is the subnet on that
> link ?

What do you mean? How can I know this? Actually, these entries are
automatically added when configuring the interfaces with ifconfig. As
you may have noted, I'm not an expert either on iptables nor on
routing U_U.
>
>> And this is what I get:
>>
>> dulceangustia@spike:~$ sudo iptables -vL
>
> This command only shows the default (filter) table. You want to display the
> mangle table with -t mangle.

Ok. Here it goes another test results. It seems that iptables mangling
works properly:


dulceangustia@spike:~$ sudo iptables -vL -t mangle
Chain PREROUTING (policy ACCEPT 114K packets, 93M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain INPUT (policy ACCEPT 113K packets, 92M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 130K packets, 135M bytes)
 pkts bytes target     prot opt in     out     source
destination
14946   22M TOS        tcp  --  any    any     anywhere
anywhere            tcp dpt:60301 TOS set Minimize-Delay
42073   63M TOS        tcp  --  any    any     anywhere
anywhere            tcp dpt:60302 TOS set Maximize-Throughput

Chain POSTROUTING (policy ACCEPT 130K packets, 135M bytes)
 pkts bytes target     prot opt in     out     source
destination
dulceangustia@spike:~$ sudo iperf -c 192.168.0.2 -t 1 -p 60301 -r
------------------------------------------------------------
Server listening on TCP port 60301
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 60301
TCP window size: 22.6 KByte (default)
------------------------------------------------------------
[  5] local 192.168.0.3 port 44517 connected with 192.168.0.2 port 60301
[  5]  0.0- 1.0 sec  3.20 MBytes  26.4 Mbits/sec
[  4] local 192.168.0.3 port 60301 connected with 192.168.0.2 port 38858
[  4]  0.0- 1.1 sec  12.8 MBytes  94.1 Mbits/sec
dulceangustia@spike:~$ sudo iptables -vL -t mangle
Chain PREROUTING (policy ACCEPT 125K packets, 107M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain INPUT (policy ACCEPT 124K packets, 106M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 137K packets, 139M bytes)
 pkts bytes target     prot opt in     out     source
destination
17404   26M TOS        tcp  --  any    any     anywhere
anywhere            tcp dpt:60301 TOS set Minimize-Delay
42073   63M TOS        tcp  --  any    any     anywhere
anywhere            tcp dpt:60302 TOS set Maximize-Throughput

Chain POSTROUTING (policy ACCEPT 137K packets, 139M bytes)
 pkts bytes target     prot opt in     out     source
destination
dulceangustia@spike:~$ sudo iperf -c 192.168.0.2 -t 1 -p 60302 -r
------------------------------------------------------------
Server listening on TCP port 60302
TCP window size: 85.3 KByte (default)
------------------------------------------------------------
------------------------------------------------------------
Client connecting to 192.168.0.2, TCP port 60302
TCP window size: 26.4 KByte (default)
------------------------------------------------------------
[  5] local 192.168.0.3 port 44293 connected with 192.168.0.2 port 60302
[  5]  0.0- 1.0 sec  11.6 MBytes  96.2 Mbits/sec
[  4] local 192.168.0.3 port 60302 connected with 192.168.0.2 port 39370
[  4]  0.0- 1.1 sec  12.1 MBytes  93.8 Mbits/sec
dulceangustia@spike:~$ sudo iptables -vL -t mangle
Chain PREROUTING (policy ACCEPT 138K packets, 120M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain INPUT (policy ACCEPT 137K packets, 120M bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source
destination

Chain OUTPUT (policy ACCEPT 150K packets, 152M bytes)
 pkts bytes target     prot opt in     out     source
destination
17404   26M TOS        tcp  --  any    any     anywhere
anywhere            tcp dpt:60301 TOS set Minimize-Delay
50518   76M TOS        tcp  --  any    any     anywhere
anywhere            tcp dpt:60302 TOS set Maximize-Throughput

Chain POSTROUTING (policy ACCEPT 151K packets, 152M bytes)
 pkts bytes target     prot opt in     out     source
destination


>
>> I don't know why the source IP address is not changed. May be a
>> problem of ip route and not iptables?
>
> I explained why the source address is unchanged, see above.

Ok.

>
>> Any idea about how to solve it?
>
> Either select the source address in the sender process (if you can select
> the destination port, you may be able to select the source address too) or
> use iptables SNAT.
>
>> I tried also to add a POSTROUTING SNAT rule to change the origin
>> source but it doesn't take any effect.
>
> It should work. What rules did you try ? Note that iperf shows only the
> initial source address before SNAT, not the actual address on the wire.

These are the SNAT rules I use:

# Change the source IP of outgoing iperf traffic to 60302 port
sudo iptables -t nat -A POSTROUTING -p tcp --dport 60302 -j SNAT
--to-source 192.168.0.3

# Change the source IP of outgoing iperf traffic to 60301 port
sudo iptables -t nat -A POSTROUTING -p tcp --dport 60301 -j SNAT
--to-source 192.168.0.150


Thank you so much,
Javi


> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>