Re: Help with packet marking

Re: Help with packet marking

[John Lister]

>Hi, i think that exist another way to do this (best).
>The linux kernel will select the routing tables according to the RPDB rules.

It seems to be selecting the correct route using the marks as iptables reports the correct interface in the log files.
However the packet then goes out of a different interface.

>You "cant" define a default gw into main table (for best routing treatment).

This has always worked before, the default route is in the main table (maybe not clear before) and is used so that
the box can route local packets out. Your example (below) would do the same except skip the fwmark rules

>But, you can change your RPDB to select a default gw out of main table (forced).
>
>For example:
>
>ip route del default
>
>ip rule flush
>ip rule add prio 10 table main
>ip rule add prio 50 fwmark 101 table 101
>ip rule add prio 51 fwmark 103 table 103
>ip rule add prio 100 table 101
>
>ip route flush cache
>
>Another thing is... the mangle table is processed per packet.
>So, i think you shouldnt use the NEW state to validate the netfilter marks.
>Your netfilter MARKs shall be made ​​with --set-mark 0x101 or 0x103

Yes, sorry when doing the example missed off the -m state --state NEW bit...
I still find it strange that recently packets I'd expect to be in the NEW state are ESTABLISHED. eg doing
ping blah
ping blah
results in the first outgoing packet being NEW, but the second ping is ESTABLISHED, surely this is a bug?

Thanks

John

>>  2012/3/28 John Lister<john.lister@kickstone.co.uk>:
>>  Hi, I've got a multi homed system which was all working fine until it was
>>  accidentally rebooted a couple of days ago and this is probably more LARTC
>>  but that list seems dead?
>>
>>  Anyway I'm now seeing bizarre behaviour running ubuntu 10.04 kernel
>>  2.6.32-40. Previously I would mark the packets in prerouting and then have
>>  fwmark based rules in the routing table to send them out via a separate
>>  custom routing table. Â I have the main routing table set up with a default
>>  route via one of the interfaces
>>
>>  Now I can see when debugging iptables, that the interface is set correctly
>>  and the packet is marked, but sticking a sniffer onto the interfaces shows
>>  me that the packets are coming out of the default interface. If i disable
>>  the default route I get "unreachable host". Another wierd thing is that
>>  conntrack is showing separate new connections as established, for example if
>>  I ping 1 packet and repeat the command the second ping is labelled as
>>  established which I wouldn't expect... (I've posted this issue earlier as
>>  restore-mark is working for what I'd expect to be new connections)
>>
>>  Anyway my setup
>>  interfaces
>>  eth0 : 192.168.2.7
>>  eth1 : x.37.63.74 Â  Â  Â  Â gw=x.37.63.73
>>  eth3 : x.45.115.81 Â  Â  Â gw=x.45.115.86
>>
>>  # add extra routing tables
>>  ip route add x.37.63.72/29 dev eth1 table 101
>>  ip route add 192.168.2.0/24 dev eth0 table 101
>>  ip route add default via x.37.63.73 dev eth1 table 101
>>
>>  ip route add x.45.115.80/29 dev eth3 table 103
>>  ip route add 192.168.2.0/24 dev eth0 table 103
>>  ip route add default via x.45.115.86 dev eth3 table 103
>>
>>  # add rules
>>  ip rule add fwmark 101 table 101
>>  ip rule add fwmark 103 table 103
>>
>>  # route a specific ip out for testing sent from 192.168.2.x and forwarded
>>  through this box.
>>  iptables -t mangle -A PREROUTING -d 98.207.221.49/32 -m state --state NEW -j
>>  MATCH1
>>  iptables -t mangle -A MATCH1 -j MARK --set-mark 0x1
>>
>>  iptables -t mangle -A MATCH1 -j CONNMARK --save-mark
>>
>>  iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to x.37.63.74
>>
>>
>>  The above is a simplified subset but covers the basics I think. Anyway doing
>>  ping 98.207.221.49
>>  from an internal machine using this as its gateway fails. I can see it marks
>>  the rule, does the routing which sets the outgoing interface correctly but
>>  then the packet comes out of the default interface and is then lost. The
>>  following logs are generated (cropped for readability) from which you can
>>  see it getting marked and the interface set correctly
>>
>>  PREROUTING (NEW) IN=eth0 OUT= Â SRC=192.168.2.133 DST=98.207.221.49 LEN=60
>>  TOS=0x00 PREC=0x00 TTL=128 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354
>>  PREROUTING (MARK1) IN=eth0 OUT= Â SRC=192.168.2.133 DST=98.207.221.49 LEN=60
>>  TOS=0x00 PREC=0x00 TTL=128 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354
>>  FORWARD IN=eth0 OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 TOS=0x00
>>  PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354 MARK=0x1
>>  POSTROUTING IN= OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 TOS=0x00
>>  PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354 MARK=0x1
>>  SNATTING IN= OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 TOS=0x00
>>  PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354 MARK=0x1


-- 
www.pricegoblin.co.uk

Re: Help with packet marking

[Humberto Jucá]

2012/3/29 John Lister <john.lister@kickstone.co.uk>:
> It seems to be selecting the correct route using the marks as iptables
> reports the correct interface in the log files.
> However the packet then goes out of a different interface.

Show us all firewall and routing rules (at least the main)...
   iptables -t mangle -nL -v
   ip rule ls

> This has always worked before, the default route is in the main table (maybe
> not clear before) and is used so that
> the box can route local packets out. Your example (below) would do the same
> except skip the fwmark rules

Not exactly. In my example, to skip the fwmark process the destination
address must be known by the main table. And you dont need to treat
your essential routes in alternative tables (only default gw). For
this reason,  you couldnt use a default gw in main table (*my
example*).

But, i still not sure why your setup has stopped working.

> Yes, sorry when doing the example missed off the -m state --state NEW bit...
> I still find it strange that recently packets I'd expect to be in the NEW
> state are ESTABLISHED. eg doing
> ping blah
> ping blah
> results in the first outgoing packet being NEW, but the second ping is
> ESTABLISHED, surely this is a bug?

Why you need to work with connection STATEs in firewall MARKs?

Tell me more about your configuration.
I can check your firewall confs if you open your ssh access for me
(send me account in pvt - if you like).

Re: Help with packet marking

[Jan Engelhardt]

On Thursday 2012-03-29 16:55, Humberto Jucá wrote:

>2012/3/29 John Lister <john.lister@kickstone.co.uk>:
>> It seems to be selecting the correct route using the marks as iptables
>> reports the correct interface in the log files.
>> However the packet then goes out of a different interface.
>
>Show us all firewall and routing rules (at least the main)...
>   iptables -t mangle -nL -v

Nono, don't use iptables -L. Better use iptables-save.

Re: Help with packet marking

[Humberto Jucá]

I still do not understand why your conntrack display MARK 0x1 and your
RPDB do a table lookup with 0x101 or 0x103.

Re: Help with packet marking

[John Lister]

On 29/03/2012 15:55, Humberto Jucá wrote:
> 2012/3/29 John Lister<john.lister@kickstone.co.uk>:
>> It seems to be selecting the correct route using the marks as iptables
>> reports the correct interface in the log files.
>> However the packet then goes out of a different interface.
> Show us all firewall and routing rules (at least the main)...
>     iptables -t mangle -nL -v
>     ip rule ls
I'm out of the office at the minute but will extract them later today.
>> This has always worked before, the default route is in the main table (maybe
>> not clear before) and is used so that
>> the box can route local packets out. Your example (below) would do the same
>> except skip the fwmark rules
> Not exactly. In my example, to skip the fwmark process the destination
> address must be known by the main table. And you dont need to treat
> your essential routes in alternative tables (only default gw). For
> this reason,  you couldnt use a default gw in main table (*my
> example*).
Ok, misread that and still had a default in main in my head. Makes sense now
> But, i still not sure why your setup has stopped working.
Neither do I? The only thing I can think of is a new kernel 
inadvertently installed by a colleague but without doing a reboot. As it 
all worked fine until it was rebooted (to physically move it). Also, 
oddly before I left last night, it was occasionally connecting  when 
doing some tests
>
>> Yes, sorry when doing the example missed off the -m state --state NEW bit...
>> I still find it strange that recently packets I'd expect to be in the NEW
>> state are ESTABLISHED. eg doing
>> ping blah
>> ping blah
>> results in the first outgoing packet being NEW, but the second ping is
>> ESTABLISHED, surely this is a bug?
> Why you need to work with connection STATEs in firewall MARKs?
I guess I don't need to, I wanted to only mark new connections and the 
use save-mark and restore-mark to mark further packets. The plan is that 
each new connection is marked using the statisic module and routed based 
on the mark.

It still seems like a bug that subsequent independent connections are 
labelled as ESTABLISHED?
> Tell me more about your configuration.
> I can check your firewall confs if you open your ssh access for me
> (send me account in pvt - if you like).
I may well do if I can't sort it quickly

-- 
www.pricegoblin.co.uk

Re: Help with packet marking

[John Lister]

Thanks to all those who offered help. I have everything working now. Not 
entirely sure what happened. 2 things of note:

- I upgraded the kernel version  from 2.6.31-23 to 2.6.32-40
- I was setting /proc/sys/net/ipv4/conf/all/rp_filter to 0 which worked 
previously, however it would seem that in the newer kernels I have to 
also set all the interface values as well. I'm using multiple IPs per 
nic and not setting this causes the return packets to get lost in some 
instances.

The kernel change seemed to fix the packets going out of the wrong 
interface problem although I can't test this any more unfortunately - 
which does seem very odd. But maybe the 2 in combination caused it?

Thanks again

John


On 29/03/2012 15:55, Humberto Jucá wrote:
> 2012/3/29 John Lister<john.lister@kickstone.co.uk>:
>> It seems to be selecting the correct route using the marks as iptables
>> reports the correct interface in the log files.
>> However the packet then goes out of a different interface.
> Show us all firewall and routing rules (at least the main)...
>     iptables -t mangle -nL -v
>     ip rule ls
>
>> This has always worked before, the default route is in the main table (maybe
>> not clear before) and is used so that
>> the box can route local packets out. Your example (below) would do the same
>> except skip the fwmark rules
> Not exactly. In my example, to skip the fwmark process the destination
> address must be known by the main table. And you dont need to treat
> your essential routes in alternative tables (only default gw). For
> this reason,  you couldnt use a default gw in main table (*my
> example*).
>
> But, i still not sure why your setup has stopped working.
>
>> Yes, sorry when doing the example missed off the -m state --state NEW bit...
>> I still find it strange that recently packets I'd expect to be in the NEW
>> state are ESTABLISHED. eg doing
>> ping blah
>> ping blah
>> results in the first outgoing packet being NEW, but the second ping is
>> ESTABLISHED, surely this is a bug?
> Why you need to work with connection STATEs in firewall MARKs?
>
> Tell me more about your configuration.
> I can check your firewall confs if you open your ssh access for me
> (send me account in pvt - if you like).


-- 
Get the PriceGoblin Browser Addon
www.pricegoblin.co.uk

Help with packet marking

[John Lister]

Hi, I've got a multi homed system which was all working fine until it 
was accidentally rebooted a couple of days ago and this is probably more 
LARTC but that list seems dead?

Anyway I'm now seeing bizarre behaviour running ubuntu 10.04 kernel 
2.6.32-40. Previously I would mark the packets in prerouting and then 
have fwmark based rules in the routing table to send them out via a 
separate custom routing table.  I have the main routing table set up 
with a default route via one of the interfaces

Now I can see when debugging iptables, that the interface is set 
correctly and the packet is marked, but sticking a sniffer onto the 
interfaces shows me that the packets are coming out of the default 
interface. If i disable the default route I get "unreachable host". 
Another wierd thing is that conntrack is showing separate new 
connections as established, for example if I ping 1 packet and repeat 
the command the second ping is labelled as established which I wouldn't 
expect... (I've posted this issue earlier as restore-mark is working for 
what I'd expect to be new connections)

Anyway my setup
interfaces
eth0 : 192.168.2.7
eth1 : x.37.63.74        gw=x.37.63.73
eth3 : x.45.115.81      gw=x.45.115.86

# add extra routing tables
ip route add x.37.63.72/29 dev eth1 table 101
ip route add 192.168.2.0/24 dev eth0 table 101
ip route add default via x.37.63.73 dev eth1 table 101

ip route add x.45.115.80/29 dev eth3 table 103
ip route add 192.168.2.0/24 dev eth0 table 103
ip route add default via x.45.115.86 dev eth3 table 103

# add rules
ip rule add fwmark 101 table 101
ip rule add fwmark 103 table 103

# route a specific ip out for testing sent from 192.168.2.x and 
forwarded through this box.
iptables -t mangle -A PREROUTING -d 98.207.221.49/32 -m state --state 
NEW -j MATCH1
iptables -t mangle -A MATCH1 -j MARK --set-mark 0x1
iptables -t mangle -A MATCH1 -j CONNMARK --save-mark

iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to x.37.63.74


The above is a simplified subset but covers the basics I think. Anyway 
doing
ping 98.207.221.49
from an internal machine using this as its gateway fails. I can see it 
marks the rule, does the routing which sets the outgoing interface 
correctly but then the packet comes out of the default interface and is 
then lost. The following logs are generated (cropped for readability) 
from which you can see it getting marked and the interface set correctly

PREROUTING (NEW) IN=eth0 OUT=  SRC=192.168.2.133 DST=98.207.221.49 
LEN=60 TOS=0x00 PREC=0x00 TTL=128 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 
SEQ=1354
PREROUTING (MARK1) IN=eth0 OUT=  SRC=192.168.2.133 DST=98.207.221.49 
LEN=60 TOS=0x00 PREC=0x00 TTL=128 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 
SEQ=1354
FORWARD IN=eth0 OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 
TOS=0x00 PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 
SEQ=1354 MARK=0x1
POSTROUTING IN= OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 
TOS=0x00 PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 
SEQ=1354 MARK=0x1
SNATTING IN= OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 
TOS=0x00 PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 
SEQ=1354 MARK=0x1

anyone got an idea what is going on?

Thanks

John










-- 
Get the PriceGoblin Browser Addon
www.pricegoblin.co.uk

Re: Help with packet marking

[Humberto Jucá]

Hi, i think that exist another way to do this (best).
The linux kernel will select the routing tables according to the RPDB rules.

You "cant" define a default gw into main table (for best routing treatment).
But, you can change your RPDB to select a default gw out of main table (forced).

For example:

ip route del default

ip rule flush
ip rule add prio 10 table main
ip rule add prio 50 fwmark 101 table 101
ip rule add prio 51 fwmark 103 table 103
ip rule add prio 100 table 101

ip route flush cache

Another thing is... the mangle table is processed per packet.
So, i think you shouldnt use the NEW state to validate the netfilter marks.
Your netfilter MARKs shall be made ​​with --set-mark 0x101 or 0x103



2012/3/28 John Lister <john.lister@kickstone.co.uk>:
> Hi, I've got a multi homed system which was all working fine until it was
> accidentally rebooted a couple of days ago and this is probably more LARTC
> but that list seems dead?
>
> Anyway I'm now seeing bizarre behaviour running ubuntu 10.04 kernel
> 2.6.32-40. Previously I would mark the packets in prerouting and then have
> fwmark based rules in the routing table to send them out via a separate
> custom routing table.  I have the main routing table set up with a default
> route via one of the interfaces
>
> Now I can see when debugging iptables, that the interface is set correctly
> and the packet is marked, but sticking a sniffer onto the interfaces shows
> me that the packets are coming out of the default interface. If i disable
> the default route I get "unreachable host". Another wierd thing is that
> conntrack is showing separate new connections as established, for example if
> I ping 1 packet and repeat the command the second ping is labelled as
> established which I wouldn't expect... (I've posted this issue earlier as
> restore-mark is working for what I'd expect to be new connections)
>
> Anyway my setup
> interfaces
> eth0 : 192.168.2.7
> eth1 : x.37.63.74        gw=x.37.63.73
> eth3 : x.45.115.81      gw=x.45.115.86
>
> # add extra routing tables
> ip route add x.37.63.72/29 dev eth1 table 101
> ip route add 192.168.2.0/24 dev eth0 table 101
> ip route add default via x.37.63.73 dev eth1 table 101
>
> ip route add x.45.115.80/29 dev eth3 table 103
> ip route add 192.168.2.0/24 dev eth0 table 103
> ip route add default via x.45.115.86 dev eth3 table 103
>
> # add rules
> ip rule add fwmark 101 table 101
> ip rule add fwmark 103 table 103
>
> # route a specific ip out for testing sent from 192.168.2.x and forwarded
> through this box.
> iptables -t mangle -A PREROUTING -d 98.207.221.49/32 -m state --state NEW -j
> MATCH1
> iptables -t mangle -A MATCH1 -j MARK --set-mark 0x1
> iptables -t mangle -A MATCH1 -j CONNMARK --save-mark
>
> iptables -t nat -A POSTROUTING -o eth1 -j SNAT --to x.37.63.74
>
>
> The above is a simplified subset but covers the basics I think. Anyway doing
> ping 98.207.221.49
> from an internal machine using this as its gateway fails. I can see it marks
> the rule, does the routing which sets the outgoing interface correctly but
> then the packet comes out of the default interface and is then lost. The
> following logs are generated (cropped for readability) from which you can
> see it getting marked and the interface set correctly
>
> PREROUTING (NEW) IN=eth0 OUT=  SRC=192.168.2.133 DST=98.207.221.49 LEN=60
> TOS=0x00 PREC=0x00 TTL=128 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354
> PREROUTING (MARK1) IN=eth0 OUT=  SRC=192.168.2.133 DST=98.207.221.49 LEN=60
> TOS=0x00 PREC=0x00 TTL=128 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354
> FORWARD IN=eth0 OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 TOS=0x00
> PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354 MARK=0x1
> POSTROUTING IN= OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 TOS=0x00
> PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354 MARK=0x1
> SNATTING IN= OUT=eth1 SRC=192.168.2.133 DST=98.207.221.49 LEN=60 TOS=0x00
> PREC=0x00 TTL=127 ID=9452 PROTO=ICMP TYPE=8 CODE=0 ID=1 SEQ=1354 MARK=0x1
>
> anyone got an idea what is going on?
>
> Thanks
>
> John