fwmark iptables/ip routing interaction question

fwmark iptables/ip routing interaction question

[pthyseba]

Hello,

After reading docs on iptables and advanced linux routing using iproute2,
I'm left with the following question:

I have succesfully setup a routing policy on Red Hat Enterprise Linux 4
where one type of locally generated traffic is routed to a gateway
different from the default gateway. I have done this straight from the
book:

add a MARK target for these packets in the OUTPUT chain of table "mangle"
add a routing rule for marked packets to a different routing table
add a default gateway to the new routing table.


This setup works as expected: the targeted packets are routed to the
different gateway. However, my question is: why does this work?

After studying the iptables block diagrams, it seems that the packet
travels through the OUTPUT chains AFTER "routing". I assumed that
"routing" implied the lookup of the correct routing table and then
selection of the correct rule in that table, but putting the iptables and
my definition of "routing" together, it would seem that packets only get
marked (the MARK target is in the output chain) after "routing" has been
done (and I assumed filtering packets using the fwmark filter was done in
that step). Obviously, I'm misunderstanding something, as packets that get
marked in the OUTPUT chain do get routed correctly (meaning, have their
mark set when routing is looking for the correct routing table).

What's wrong in my understanding of the interaction between iptables and
"routing" ?

Thanks,
Pieter

Re: fwmark iptables/ip routing interaction question

[Jan Engelhardt]

On Wednesday 2008-05-07 16:36, pthyseba@intec.ugent.be wrote:
>
>This setup works as expected: the targeted packets are routed to the
>different gateway. However, my question is: why does this work?

Because we expect it to :)

>After studying the iptables block diagrams, it seems that the packet
>travels through the OUTPUT chains AFTER "routing". I assumed that
>"routing" implied the lookup of the correct routing table and then
>selection of the correct rule in that table, but putting the iptables and
>my definition of "routing" together, it would seem that packets only get
>marked (the MARK target is in the output chain) after "routing" has been
>done (and I assumed filtering packets using the fwmark filter was done in
>that step). Obviously, I'm misunderstanding something, as packets that get
>marked in the OUTPUT chain do get routed correctly (meaning, have their
>mark set when routing is looking for the correct routing table).

Correct, there is an extra reroute done in the output path
(see http://jengelh.medozas.de/images/nf-packet-flow.png )

Re: fwmark iptables/ip routing interaction question

[Alexei Ustyuzhaninov]

Jan Engelhardt wrote:
> On Wednesday 2008-05-07 16:36, pthyseba@intec.ugent.be wrote:
>> This setup works as expected: the targeted packets are routed to the
>> different gateway. However, my question is: why does this work?
> 
> Because we expect it to :)
> 
>> After studying the iptables block diagrams, it seems that the packet
>> travels through the OUTPUT chains AFTER "routing". I assumed that
>> "routing" implied the lookup of the correct routing table and then
>> selection of the correct rule in that table, but putting the iptables and
>> my definition of "routing" together, it would seem that packets only get
>> marked (the MARK target is in the output chain) after "routing" has been
>> done (and I assumed filtering packets using the fwmark filter was done in
>> that step). Obviously, I'm misunderstanding something, as packets that get
>> marked in the OUTPUT chain do get routed correctly (meaning, have their
>> mark set when routing is looking for the correct routing table).
> 
> Correct, there is an extra reroute done in the output path
> (see http://jengelh.medozas.de/images/nf-packet-flow.png )

Jan, could you explain please which code does this extra rerouting: 
iproute2 or netfilter? I have a weird with the same scenario: after 
rerouting a packet goes out through another interface but keeps the 
source address of the original interface.

-- 
Alexei

Re: fwmark iptables/ip routing interaction question

[Jan Engelhardt]

On Sunday 2008-05-11 18:23, Alexei Ustyuzhaninov wrote:
>
> Jan, could you explain please which code does this extra rerouting: iproute2 or
> netfilter?

Netfilter of course.
grep -r route_me_harder net/ipv4/netfilter/

> I have a weird with the same scenario: after rerouting a packet goes
> out through another interface but keeps the source address of the original
> interface.

Yes that is ... how the code is currently written it seems.

Re: fwmark iptables/ip routing interaction question

[Alexei Ustyuzhaninov]

Jan Engelhardt wrote:
> On Sunday 2008-05-11 18:23, Alexei Ustyuzhaninov wrote:
>> Jan, could you explain please which code does this extra rerouting: iproute2 or
>> netfilter?
> 
> Netfilter of course.
> grep -r route_me_harder net/ipv4/netfilter/
> 
>> I have a weird with the same scenario: after rerouting a packet goes
>> out through another interface but keeps the source address of the original
>> interface.
> 
> Yes that is ... how the code is currently written it seems.

Well, maybe you can suggest how to work around this? :) I tried snat, 
but this solution seems not to work.

-- 
Alexei

Re: fwmark iptables/ip routing interaction question

[Pascal Hambourg]

Hello,

Alexei Ustyuzhaninov a écrit :
>>
>>> I have a weird with the same scenario: after rerouting a packet goes
>>> out through another interface but keeps the source address of the 
>>> original interface.
>>
>> Yes that is ... how the code is currently written it seems.

The source address is chosen before the packet is sent to the OUTPUT 
chains, and cannot be changed by the rerouting code, this would break 
things such as connection tracking. It requires stateful NAT so the 
correct original address is put back in replies. By the way, DNAT in the 
OUTPUT chain used to implicitly change the source address when the new 
destination address caused the output interface to change until kernel 
2.6.11.

> Well, maybe you can suggest how to work around this? :) I tried snat, 
> but this solution seems not to work.

SNAT should work on packets creating a new connection (i.e. in the state 
NEW).

Re: fwmark iptables/ip routing interaction question

[Alexei Ustyuzhaninov]

Pascal Hambourg wrote:
> Hello,
> 
> Alexei Ustyuzhaninov a écrit :
>>>
>>>> I have a weird with the same scenario: after rerouting a packet goes
>>>> out through another interface but keeps the source address of the 
>>>> original interface.
>>>
>>> Yes that is ... how the code is currently written it seems.
> 
> The source address is chosen before the packet is sent to the OUTPUT 
> chains, and cannot be changed by the rerouting code, this would break 
> things such as connection tracking. It requires stateful NAT so the 
> correct original address is put back in replies. By the way, DNAT in the 
> OUTPUT chain used to implicitly change the source address when the new 
> destination address caused the output interface to change until kernel 
> 2.6.11.

Hm. That's not clear to me as I don't know the destination address in 
advance (and don't want to change it). Anyhow this solution is obsolete 
as I see.

>> Well, maybe you can suggest how to work around this? :) I tried snat, 
>> but this solution seems not to work.
> 
> SNAT should work on packets creating a new connection (i.e. in the state 
> NEW).

Yes, really! The SYN packet goes out through the right interface with 
the right source address, SYN/ACK comes back and that's all, nothing 
will happen more.

Anyhow  isn't this problem unresolvable in general? I just want a simple 
thing: to send mail via one provider and all other traffic via the other 
provider,

-- 
Thanks,
Alexei

Re: fwmark iptables/ip routing interaction question

[Pascal Hambourg]

Alexei Ustyuzhaninov a écrit :
> Pascal Hambourg wrote:
>>
>> SNAT should work on packets creating a new connection (i.e. in the 
>> state NEW).
> 
> Yes, really! The SYN packet goes out through the right interface with 
> the right source address, SYN/ACK comes back and that's all, nothing 
> will happen more.

This looks like a filtering issue causing the reply packet to be 
dropped. Check your iptables 'filter' rules and that source validation 
by reversed path is disabled for that interface 
(/proc/sys/net/ipv4/conf/<interface>/rp_filter=0).

> I just want a simple thing: to send mail via one provider and all other
> traffic via the other provider

You may be able to specify the desired source address for outgoing 
connections if your mail application allows it.

Re: fwmark iptables/ip routing interaction question

[Alexei Ustyuzhaninov]

Pascal Hambourg wrote:
> Alexei Ustyuzhaninov a écrit :
>> Pascal Hambourg wrote:
>>>
>>> SNAT should work on packets creating a new connection (i.e. in the 
>>> state NEW).
>>
>> Yes, really! The SYN packet goes out through the right interface with 
>> the right source address, SYN/ACK comes back and that's all, nothing 
>> will happen more.
> 
> This looks like a filtering issue causing the reply packet to be 
> dropped. Check your iptables 'filter' rules and that source validation 
> by reversed path is disabled for that interface 
> (/proc/sys/net/ipv4/conf/<interface>/rp_filter=0).

Yes, rp_filter was the issue indeed. Thank you very much.

>> I just want a simple thing: to send mail via one provider and all other
>> traffic via the other provider
> 
> You may be able to specify the desired source address for outgoing 
> connections if your mail application allows it.


No, of course the mail application doesn't bother about source addresses 
and IP routing. I believe it operates at different level of ISO model. :)

-- 
Thanks again,
Alexei

Re: fwmark iptables/ip routing interaction question

[Pascal Hambourg]

Alexei Ustyuzhaninov a écrit :
> Pascal Hambourg wrote:
>>
>> You may be able to specify the desired source address for outgoing 
>> connections if your mail application allows it.
> 
> No, of course the mail application doesn't bother about source addresses 
> and IP routing. I believe it operates at different level of ISO model.

An application can specify the source address to use when making 
outgoing connections. Common applications such as telnet or ping allow 
it as an option. The mail software exim allows it too.