Success routing mark'ed packets - but still confused why it didn't work the first time....

Success routing mark'ed packets - but still confused why it didn't work the first time....

[Eric B.]

Okay - after an unbelievable number of hours struggling with this, I have
finally managed to get my server working; responding to packets using 2
different gateways based on 2 different virtual ips.  However, I don't
understand why my original concept wasn't working.

Step 0) Create a default route in the main routing table for all "regular"
packets
Step 1) Create an new iproute2 routing table with a default gw and an
associated ip rule that redirects all "mark"ed packets to that table/route.
Step 2) Marking the packets:
Originally, I tried mark'ing packets in the mangle Prerouting table destined
for my machine's virtual ip.  I was under the impression that any response
to those packets would maintain the mark and hence be routed through the
appropriate routing table created in step 1.  However, this didn't work.
And without any way of "seeing" whether the outgoing packets were marked, I
couldn't tell why they weren't being routed properly (BTW - is there a way
to "see" the mark on the packet in the log?)

My solution was to use the mangle Output table to mark all the outgoing
packets with their source being the virtual ip.  Once I did that, success.
My outgoing packets are properly redirected out the appropriate gateways.


However my question now is the root of my confusion.  If a packet is mark'ed
in the Preroute mangle table, is that mark not supposed to be maintained
throughout the life of the packet, including the machine's response to that
packet?  If not, there is a lot of documentation that indicates as much,
including several howtos and guides that seem to indicate it.  For example,
http://linux-ip.net/html/linux-ip.html#adv-multi-internet-outbound indicates
to set the mark in the mangle preroute table, and then to add an ip rule.
However, as much as I tried, that soln just didn't work.

Can anyhow help clear this up a little?  When/where/why does that mark get
cleared/reset?

Thanks,

Eric

Re: Success routing mark'ed packets - but still confused why it didn't work the first time....

[Jan Engelhardt]

On Wednesday 2008-04-09 00:06, Eric B. wrote:

>Step 2) Marking the packets:
>Originally, I tried mark'ing packets in the mangle Prerouting table destined
>for my machine's virtual ip.  I was under the impression that any response
>to those packets would maintain the mark and hence be routed through the
>appropriate routing table created in step 1.  However, this didn't work.

No, use CONNMARK as shown in http://dev.computergmbh.de/NF-Cookbook.tx

>And without any way of "seeing" whether the outgoing packets were marked, I
>couldn't tell why they weren't being routed properly (BTW - is there a way
>to "see" the mark on the packet in the log?)

LOGMARK :p

>My solution was to use the mangle Output table to mark all the outgoing
>packets with their source being the virtual ip.  Once I did that, success.
>My outgoing packets are properly redirected out the appropriate gateways.

Make sure you do not set a mark in an input/prerouting
chain so that it accidentally hits the routing rule
because that would mean incoming packets are diverted
before they reach the machine.

>However my question now is the root of my confusion.  If a packet is mark'ed
>in the Preroute mangle table, is that mark not supposed to be maintained
>throughout the life of the packet,

The mark is maintained during the life of the packet (until you change it
of course).

> including the machine's response to that
>packet?

Response is a new packet. Only the ctmark (connection mark)
can "survive" here.

Re: Success routing mark'ed packets - but still confused why it didn't work the first time....

[Eric B.]

"Jan Engelhardt" <jengelh@computergmbh.de> wrote in message 
news:alpine.LNX.1.10.0804090612580.2229@fbirervta.pbzchgretzou.qr...
>
> On Wednesday 2008-04-09 00:06, Eric B. wrote:
>>And without any way of "seeing" whether the outgoing packets were marked, 
>>I
>>couldn't tell why they weren't being routed properly (BTW - is there a way
>>to "see" the mark on the packet in the log?)
>
> LOGMARK :p

Ah yes - back to the fun part about having to work on a fixed disto - ie: 
RHEL4. :)


>>My solution was to use the mangle Output table to mark all the outgoing
>>packets with their source being the virtual ip.  Once I did that, success.
>>My outgoing packets are properly redirected out the appropriate gateways.
>
> Make sure you do not set a mark in an input/prerouting
> chain so that it accidentally hits the routing rule
> because that would mean incoming packets are diverted
> before they reach the machine.

But isn't that exactly what you are doing in your NF-Cookbook.txt?  You 
have:
-t mangle -A PREROUTING -j CONNMARK --restore-mark
Isn't that setting the mark in the prerouting table?  Or do you do it that 
way since the local routing table is the first table, and any packets 
destined for this machine will be accepted anyhow?


>>However my question now is the root of my confusion.  If a packet is 
>>mark'ed
>>in the Preroute mangle table, is that mark not supposed to be maintained
>>throughout the life of the packet,
>
> The mark is maintained during the life of the packet (until you change it
> of course).
>
>> including the machine's response to that
>>packet?
>
> Response is a new packet. Only the ctmark (connection mark)
> can "survive" here.

Do the ip rules based on the fwmark work on the individual packet's mark 
value or the conntrack mark, or both?

Thanks for the clarification!

Eric

Re: Success routing mark'ed packets - but still confused why it didn't work the first time....

[Eric B.]

"Eric B." <ebenze@hotmail.com> wrote in message 
news:ftj951$3cv$1@ger.gmane.org...
> "Jan Engelhardt" <jengelh@computergmbh.de> wrote in message 
> news:alpine.LNX.1.10.0804090612580.2229@fbirervta.pbzchgretzou.qr...

> Do the ip rules based on the fwmark work on the individual packet's mark 
> value or the conntrack mark, or both?

If both (as what it seems), why do you bother with the --restore-mark in the 
PREROUTING chain, if the connection is already marked anyhow?  And if not, 
then why do you not have a --restore-mark in both the FORWARD and OUTPUT 
chains?

Thanks,

Eric

Re: Success routing mark'ed packets - but still confused why it didn't work the first time....

[Jan Engelhardt]

On Wednesday 2008-04-09 22:32, Eric B. wrote:
>>
>> Make sure you do not set a mark in an input/prerouting
>> chain so that it accidentally hits the routing rule
>> because that would mean incoming packets are diverted
>> before they reach the machine.
>
>But isn't that exactly what you are doing in your NF-Cookbook.txt?  You 
>have:
>-t mangle -A PREROUTING -j CONNMARK --restore-mark
>Isn't that setting the mark in the prerouting table?

Yes...

>Or do you do it that 
>way since the local routing table is the first table, and any packets 
>destined for this machine will be accepted anyhow?

On the very first packet, the incoming interface (or some other kind
of parameter perhaps) is recorded by setting a connection mark. This
connection mark is then copied to all packets in a given direction so
as to route them into the interface they first one came from, for
example. Because the connection mark is only set once, it can be
assured that all packets will get the same mark.

>> The mark is maintained during the life of the packet (until you change it
>> of course).
>>
>>> including the machine's response to that
>>>packet?
>>
>> Response is a new packet. Only the ctmark (connection mark)
>> can "survive" here.
>
>Do the ip rules based on the fwmark work on the individual packet's mark 
>value or the conntrack mark, or both?

routing rules work on packet mark ("nfmark" "fwmark"), whch is
why --restore-mark is needed.

Re: Success routing mark'ed packets - but still confused why it didn't work the first time....

[Eric B.]

"Jan Engelhardt" <jengelh@computergmbh.de> wrote in message 
news:alpine.LNX.1.10.0804092310390.17930@fbirervta.pbzchgretzou.qr...
>>Do the ip rules based on the fwmark work on the individual packet's mark
>>value or the conntrack mark, or both?
>
> routing rules work on packet mark ("nfmark" "fwmark"), whch is
> why --restore-mark is needed.

But then, do you not need a --restore-mark in both the FORWARD and OUTPUT 
chains?

-t mangle -A FORWARD -m connmark --mark 0 -m conntrack --ctstate NEW -j 
prefout
	-t mangle -A OUTPUT -m connmark --mark 0 -m conntrack --ctstate NEW -j 
prefout

Or are you simply marking the connection and expecting the system to 
randomly choose whether to output on ppp0 or ppp1?

Thanks,

Eric