Routing locally generated traffic on fwmark

Routing locally generated traffic on fwmark

[Andrew Beverley]

Hi,

I'd like to route locally generated traffic via a particular interface
based on its mark value.

From what I have researched, this is theoretically possible and lots of
people have tried it, but nobody has got it working.

Here's my rules:

# Mark the packets
iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800

# Route the marked packets via routing table T2:
ip rule add fwmark 0x800/0xffff table T2

# Force T2 packets out of the interface ppp1
ip route add table T2 default dev ppp1 via 94.30.127.76

# Flush the cache, just in case
ip route flush cache

However, the packets still go out of the default route (ppp0).

cwdwr:~# ip rule
0:	from all lookup local 
32765:	from all fwmark 0x800/0xffff lookup T2 
32766:	from all lookup main 
32767:	from all lookup default 

cwdwr:~# ip route show table T2
default via 94.30.127.76 dev ppp1 

Any ideas? Should this be possible?

Thanks,

Andy

Re: Routing locally generated traffic on fwmark

[Andrew Beverley]

On Wed, 2011-09-28 at 23:20 +0100, Andrew Beverley wrote:
> Hi,
> 
> I'd like to route locally generated traffic via a particular interface
> based on its mark value.
> 
> From what I have researched, this is theoretically possible and lots of
> people have tried it, but nobody has got it working.
> 
> Here's my rules:
> 
> # Mark the packets
> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
> 
> # Route the marked packets via routing table T2:
> ip rule add fwmark 0x800/0xffff table T2
> 
> # Force T2 packets out of the interface ppp1
> ip route add table T2 default dev ppp1 via 94.30.127.76
> 
> # Flush the cache, just in case
> ip route flush cache
> 
> However, the packets still go out of the default route (ppp0).

I've also added the following, which makes no difference:

iptables -t nat -A POSTROUTING -o ppp1 \
	-j SNAT --to-source 109.224.134.110

And I've done a test with:

ip rule add to 89.16.176.81 table T2

which *does* work.

So, I assume the problem is that the packet is marked too late to affect
the routing. Looking at the packet flow diagram[1] though, there should
be a re-route check after the mangle table, which should re-route if a
packet's mark has changed. Does this feature need enabling?

Andy

[1] http://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg

Re: Routing locally generated traffic on fwmark

[Pandu Poluan]

On Thu, Sep 29, 2011 at 13:51, Andrew Beverley <andy@andybev.com> wrote:
> On Wed, 2011-09-28 at 23:20 +0100, Andrew Beverley wrote:
>> Hi,
>>
>> I'd like to route locally generated traffic via a particular interface
>> based on its mark value.
>>
>> From what I have researched, this is theoretically possible and lots of
>> people have tried it, but nobody has got it working.
>>
>> Here's my rules:
>>
>> # Mark the packets
>> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
>>
>> # Route the marked packets via routing table T2:
>> ip rule add fwmark 0x800/0xffff table T2
>>
>> # Force T2 packets out of the interface ppp1
>> ip route add table T2 default dev ppp1 via 94.30.127.76
>>
>> # Flush the cache, just in case
>> ip route flush cache
>>
>> However, the packets still go out of the default route (ppp0).
>
> I've also added the following, which makes no difference:
>
> iptables -t nat -A POSTROUTING -o ppp1 \
>        -j SNAT --to-source 109.224.134.110
>
> And I've done a test with:
>
> ip rule add to 89.16.176.81 table T2
>
> which *does* work.
>
> So, I assume the problem is that the packet is marked too late to affect
> the routing. Looking at the packet flow diagram[1] though, there should
> be a re-route check after the mangle table, which should re-route if a
> packet's mark has changed. Does this feature need enabling?
>
> Andy
>
> [1] http://upload.wikimedia.org/wikipedia/commons/3/37/Netfilter-packet-flow.svg
>

Can you post the complete table, i.e., the output of iptables-save ?

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: [SOLVED] Routing locally generated traffic on fwmark

[Andrew Beverley]

On Thu, 2011-09-29 at 14:32 +0700, Pandu Poluan wrote:
> On Thu, Sep 29, 2011 at 13:51, Andrew Beverley <andy@andybev.com> wrote:
> > On Wed, 2011-09-28 at 23:20 +0100, Andrew Beverley wrote:
> >> Hi,
> >>
> >> I'd like to route locally generated traffic via a particular interface
> >> based on its mark value.
> >>
> >> From what I have researched, this is theoretically possible and lots of
> >> people have tried it, but nobody has got it working.
> >>
> >> Here's my rules:
> >>
> >> # Mark the packets
> >> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
> >>
> >> # Route the marked packets via routing table T2:
> >> ip rule add fwmark 0x800/0xffff table T2
> >>
> >> # Force T2 packets out of the interface ppp1
> >> ip route add table T2 default dev ppp1 via 94.30.127.76
> >>
> >> # Flush the cache, just in case
> >> ip route flush cache
> >>
> >> However, the packets still go out of the default route (ppp0).
> >
> > I've also added the following, which makes no difference:
> >
> > iptables -t nat -A POSTROUTING -o ppp1 \
> >        -j SNAT --to-source 109.224.134.110
> >
> >
> 
> Can you post the complete table, i.e., the output of iptables-save ?
> 

Thanks for that. After I added the SNAT rule, I forgot to remove an
existing earlier rule that was stopping the packets being marked. Your
email reminded me!

So, the reason it wasn't working for me was the missing SNAT rule after
all. It now works correctly.

Thanks,

Andy

Re: [SOLVED] Routing locally generated traffic on fwmark

[Pandu Poluan]

On Thu, Sep 29, 2011 at 14:53, Andrew Beverley <andy@andybev.com> wrote:
> On Thu, 2011-09-29 at 14:32 +0700, Pandu Poluan wrote:
>> On Thu, Sep 29, 2011 at 13:51, Andrew Beverley <andy@andybev.com> wrote:
>> > On Wed, 2011-09-28 at 23:20 +0100, Andrew Beverley wrote:
>> >> Hi,
>> >>
>> >> I'd like to route locally generated traffic via a particular interface
>> >> based on its mark value.
>> >>
>> >> From what I have researched, this is theoretically possible and lots of
>> >> people have tried it, but nobody has got it working.
>> >>
>> >> Here's my rules:
>> >>
>> >> # Mark the packets
>> >> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
>> >>
>> >> # Route the marked packets via routing table T2:
>> >> ip rule add fwmark 0x800/0xffff table T2
>> >>
>> >> # Force T2 packets out of the interface ppp1
>> >> ip route add table T2 default dev ppp1 via 94.30.127.76
>> >>
>> >> # Flush the cache, just in case
>> >> ip route flush cache
>> >>
>> >> However, the packets still go out of the default route (ppp0).
>> >
>> > I've also added the following, which makes no difference:
>> >
>> > iptables -t nat -A POSTROUTING -o ppp1 \
>> >        -j SNAT --to-source 109.224.134.110
>> >
>> >
>>
>> Can you post the complete table, i.e., the output of iptables-save ?
>>
>
> Thanks for that. After I added the SNAT rule, I forgot to remove an
> existing earlier rule that was stopping the packets being marked. Your
> email reminded me!
>
> So, the reason it wasn't working for me was the missing SNAT rule after
> all. It now works correctly.
>
> Thanks,
>

You're welcome.

That's why I now no longer write iptables commands directly on the
shell. I keep my firewall rules in a file /etc/opt/firewall, and if I
need to add new rules, I just do: `vi /etc/opt/firewall &&
iptables-restore < /etc/opt/firewall`

(Of course, to seed the file I'd do `iptables-save > /etc/opt/firewall` )

This has the added benefit of allowing me to document all firewall
changes by doing `hg commit` followed by `hg push` to a local
Mercurial repository.

(The reason why I put the rules in /etc/opt instead of /etc is so that
I don't have to create an .hgignore file)

Rgds,
-- 
FdS Pandu E Poluan
~ IT Optimizer ~

 • LOPSA Member #15248
 • Blog : http://pepoluan.tumblr.com
 • Linked-In : http://id.linkedin.com/in/pepoluan

Re: Routing locally generated traffic on fwmark

[Jan Engelhardt]

On Thursday 2011-09-29 08:51, Andrew Beverley wrote:
>> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
>> ip rule add fwmark 0x800/0xffff table T2
>> ip route add table T2 default dev ppp1 via 94.30.127.76
>
>I've also added the following, which makes no difference:
>
>iptables -t nat -A POSTROUTING -o ppp1 \
>	-j SNAT --to-source 109.224.134.110

Of course it makes no difference, because SNAT is applied after routing.
("POST" "ROUTING", see?)

>So, I assume the problem is that the packet is marked too late to affect
>the routing.
>Looking at the packet flow diagram[1] though, there should
>be a re-route check after the mangle table, which should re-route if a
>packet's mark has changed. Does this feature need enabling?

mangle is the right place; in its code you will find

        ret = ipt_do_table(skb, NF_INET_LOCAL_OUT, NULL, out,
                           dev_net(out)->ipv4.iptable_mangle);
        /* Reroute for ANY change. */
        if (ret != NF_DROP && ret != NF_STOLEN) {
                iph = ip_hdr(skb);
                if (iph->saddr != saddr ||
                    iph->daddr != daddr ||
                    skb->mark != mark ||  
                    iph->tos != tos)
                        if (ip_route_me_harder(skb, RTN_UNSPEC))
                                ret = NF_DROP;

Re: Routing locally generated traffic on fwmark

[Andrew Beverley]

On Thu, 2011-09-29 at 12:28 +0200, Jan Engelhardt wrote:
> On Thursday 2011-09-29 08:51, Andrew Beverley wrote:
> >> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
> >> ip rule add fwmark 0x800/0xffff table T2
> >> ip route add table T2 default dev ppp1 via 94.30.127.76
> >
> >I've also added the following, which makes no difference:
> >
> >iptables -t nat -A POSTROUTING -o ppp1 \
> >	-j SNAT --to-source 109.224.134.110
> 
> Of course it makes no difference, because SNAT is applied after routing.
> ("POST" "ROUTING", see?)

Yes, but in my case the SNAT still needed applying. The problem was that
although the packets were being routed via the second interface, they
were still being sent from the original IP address of the first
interface. Therefore, packets were being returned to the first
interface, making it look as if the second interface wasn't being used.

> 
> >So, I assume the problem is that the packet is marked too late to affect
> >the routing.
> >Looking at the packet flow diagram[1] though, there should
> >be a re-route check after the mangle table, which should re-route if a
> >packet's mark has changed. Does this feature need enabling?
> 
> mangle is the right place; in its code you will find
> 
>         ret = ipt_do_table(skb, NF_INET_LOCAL_OUT, NULL, out,
>                            dev_net(out)->ipv4.iptable_mangle);
>         /* Reroute for ANY change. */
>         if (ret != NF_DROP && ret != NF_STOLEN) {
>                 iph = ip_hdr(skb);
>                 if (iph->saddr != saddr ||
>                     iph->daddr != daddr ||
>                     skb->mark != mark ||  
>                     iph->tos != tos)
>                         if (ip_route_me_harder(skb, RTN_UNSPEC))
>                                 ret = NF_DROP;

Thanks, that helps. Useful to know exactly what is going on.

Thanks for all the help.

Andy

Re: Routing locally generated traffic on fwmark

[Jan Engelhardt]

On Thursday 2011-09-29 19:28, Andrew Beverley wrote:

>On Thu, 2011-09-29 at 12:28 +0200, Jan Engelhardt wrote:
>> On Thursday 2011-09-29 08:51, Andrew Beverley wrote:
>> >> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
>> >> ip rule add fwmark 0x800/0xffff table T2
>> >> ip route add table T2 default dev ppp1 via 94.30.127.76
>> >
>> >I've also added the following, which makes no difference:
>> >
>> >iptables -t nat -A POSTROUTING -o ppp1 \
>> >	-j SNAT --to-source 109.224.134.110
>> 
>> Of course it makes no difference, because SNAT is applied after routing.
>> ("POST" "ROUTING", see?)
>
>Yes, but in my case the SNAT still needed applying. The problem was that
>although the packets were being routed via the second interface, they
>were still being sent from the original IP address of the first
>interface. Therefore, packets were being returned to the first
>interface, making it look as if the second interface wasn't being used.

Well, that's why one should use tcpdump -i ethX, rather than tcpdump -i 
any :-)

Re: Routing locally generated traffic on fwmark

[Andrew Beverley]

On Thu, 2011-09-29 at 19:35 +0200, Jan Engelhardt wrote:
> On Thursday 2011-09-29 19:28, Andrew Beverley wrote:
> 
> >On Thu, 2011-09-29 at 12:28 +0200, Jan Engelhardt wrote:
> >> On Thursday 2011-09-29 08:51, Andrew Beverley wrote:
> >> >> iptables -A OUTPUT -t mangle -d 89.16.176.81 -j MARK --set-mark 0x800
> >> >> ip rule add fwmark 0x800/0xffff table T2
> >> >> ip route add table T2 default dev ppp1 via 94.30.127.76
> >> >
> >> >I've also added the following, which makes no difference:
> >> >
> >> >iptables -t nat -A POSTROUTING -o ppp1 \
> >> >	-j SNAT --to-source 109.224.134.110
> >> 
> >> Of course it makes no difference, because SNAT is applied after routing.
> >> ("POST" "ROUTING", see?)
> >
> >Yes, but in my case the SNAT still needed applying. The problem was that
> >although the packets were being routed via the second interface, they
> >were still being sent from the original IP address of the first
> >interface. Therefore, packets were being returned to the first
> >interface, making it look as if the second interface wasn't being used.
> 
> Well, that's why one should use tcpdump -i ethX, rather than tcpdump -i 
> any :-)

Yep, I learn something every time ;-)

Re: [SOLVED] Routing locally generated traffic on fwmark

[Ed W]

On 29/09/2011 09:29, Pandu Poluan wrote:
> That's why I now no longer write iptables commands directly on the
> shell. I keep my firewall rules in a file /etc/opt/firewall, and if I
> need to add new rules, I just do: `vi /etc/opt/firewall &&
> iptables-restore < /etc/opt/firewall`
>
> (Of course, to seed the file I'd do `iptables-save > /etc/opt/firewall` )
>
> This has the added benefit of allowing me to document all firewall
> changes by doing `hg commit` followed by `hg push` to a local
> Mercurial repository.
>
> (The reason why I put the rules in /etc/opt instead of /etc is so that
> I don't have to create an .hgignore file)
>

Can I also leave a plug for shorewall for similar reasons.  It is a
fairly thin wrapper over iptables (etc), but it allows you to think at a
slightly higher level and wraps things such as setting/restoring fwmarks
and routing, breaks them out from the general access rules.

I find it picks a very nice level between firewall guis and raw editing
of iptables commands.  Give it a try.

Also it's text file based so it's very easy to track via some source
code control system

Cheers

Ed W