I need help with "CONNMARK --set-mark"

I need help with "CONNMARK --set-mark"

[Adrian Turcu]

[-- Attachment #1: Type: text/plain, Size: 4766 bytes --]

Hello list,

I experience some intermittent problems with CONNMARK chain when try to 
conserve the connection mark. Sometimes it works just fine, but sometimes it 
seems that it stops working for whatever reason. Let me explain my setup 
(it's a test environment):


outside world | ---- | eth1 Linux_box eth0 | ---- | server

The traffic to the server has to be marked in order to shape it with tc 
(iproute2). Well, I envisage that the rules in iptables will be growing fast 
for the production environment, so I took advise from someone (Wang Jiang) on 
LARTC who suggested to use CONNMARK to skip processing the long rules-list 
for each packet enters/output my Linux box (which is really cool stuff). The 
discussion was there on a different subject, but the problems I have are 
related (I believe) with CONNMARK behaviour.

So, I have the rules as follows:


# match 'n' skip already marked traffic
/sbin/iptables -t mangle -A PREROUTING -m connmark --mark 0x8000/0x8000 \
	-j CONNMARK --restore-mark --mask 0xffff
/sbin/iptables -t mangle -A PREROUTING -m connmark --mark 0x8000/0x8000 \
	-j RETURN
#
# Assign different marks to different traffic:
# 192.168.1.218/32 is my server IP address and
# the traffic refers to Ingress (Egress will be similar)

# create a class :3280x
/sbin/iptables -t mangle -N class_3280x
/sbin/iptables -t mangle -A PREROUTING -i eth1 \
	-d ! 192.168.1.218/255.255.255.255 -j class_3280x
# mark the traffic matchin this class and return 
# class :32801	(section 1)
/sbin/iptables -t mangle -A class_3280x -j CONNMARK --set-mark 0x8021/0xFFFF
/sbin/iptables -t mangle -A class_3280x -j RETURN

# create a class: 6550x
/sbin/iptables -t mangle -N class_6550x
/sbin/iptables -t mangle -A PREROUTING -i eth1 \
	-d 192.168.1.218/255.255.255.255 -j class_6550x
# mark the traffic matching this class and return
# class 65501 (section 2)
/sbin/iptables -t mangle -A class_6550x -j CONNMARK --set-mark 0xFFDD/0xFFFF
/sbin/iptables -t mangle -A class_6550x -j RETURN

# iprtoute2 tc stuff to regulate the traffic
#
# Ingress
/sbin/tc qdisc add dev eth1 handle ffff: ingress

# class :32801 (0x8021)
/sbin/tc filter add dev eth1 parent ffff: protocol ip prio 10 \
        u32 match mark 0x8021 0xffff \
        police rate 512kbit burst 512kbit mtu 1500 drop flowid :1

# class :65501  (0xFFDD)
/sbin/tc filter add dev eth1 parent ffff: protocol ip prio 10 \
        u32 match mark 0xFFDD 0xffff \
        police rate 256kbit burst 256kbit mtu 1500 drop flowid :1


Well, I generate some traffic on the network of eth1 to match the above rules. 
I can see first packets (SYN if the rules are applied before the connections 
establishes) matching the iptables rules for each class and then the flow is 
matched by the first 2 rules only (which is what I intended to do - skip the 
rules if traffic already marked). Now, I can also see the marked connections 
using

cat /proc/net/ip_conntrack

which is fine.


When this setup works the traffic matches the tc rules, no probs, and it is 
assigned the bandwidth I want. When it doesn't work... the traffic matches 
the iptables rules in the same way when it's working, I can still track the 
connections marks in /proc/net/ip_conntrack, but the tc rules are never 
applied (I can see the stats for tc intercepting the traffic but not matching 
the marks, it's like they are not there).

Some will say that this is related with iproute2 tools, but I have came across 
with this situation: instead of CONNTRACK marking, I do normal MARK and save 
the marks with CONNTRACK (just before RETURN), which means that in sections 1 
& 2 identified above (for iptables rules) I rewrite the rules like this:

/sbin/iptables -t mangle -A class_3280x -j MARK --set-mark 0x8021
/sbin/iptables -t mangle -A class_3280x -j CONNMARK --save-mark
/sbin/iptables -t mangle -A class_3280x -j RETURN

respectively,

/sbin/iptables -t mangle -A class_6550x -j MARK --set-mark 0xFFDD
/sbin/iptables -t mangle -A class_3280x -j CONNMARK --save-mark
/sbin/iptables -t mangle -A class_6550x -j RETURN

This works always for me, I have try it dozens of time with no problem, whilst 
the --set-mark in CONNTRACK works sometime 1 out of 10 tries or never. It 
looks that setting the mark per connection is not that accurate and I need to 
remark every packet if unmarked? Am I doing something wrong?

I'm using custom built kernel 2.6.11.6 and iptables 1.3.1. This rather strange 
behaviour occurs (for at least I could tell) when restarting the linux box 
and reaplying the rules or just deleting the rules and reaplying them again 
or inserting new matching rules.

Any help will be much appreciated,
Adrian

[-- Attachment #2: Type: application/pgp-signature, Size: 189 bytes --]

Re: I need help with "CONNMARK --set-mark"

[Adrian Turcu]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

I'm just trying to revive this as there was no answer till now. Maybe I'm 
lucky this time :)

Thanks,
Adrian


On Mon 11 Apr 2005 11:34, Adrian Turcu wrote:
> Hello list,
>
> I experience some intermittent problems with CONNMARK chain when try to
> conserve the connection mark. Sometimes it works just fine, but sometimes
> it seems that it stops working for whatever reason. Let me explain my setup
> (it's a test environment):
>
>
> outside world | ---- | eth1 Linux_box eth0 | ---- | server
>
> The traffic to the server has to be marked in order to shape it with tc
> (iproute2). Well, I envisage that the rules in iptables will be growing
> fast for the production environment, so I took advise from someone (Wang
> Jiang) on LARTC who suggested to use CONNMARK to skip processing the long
> rules-list for each packet enters/output my Linux box (which is really cool
> stuff). The discussion was there on a different subject, but the problems I
> have are related (I believe) with CONNMARK behaviour.
>
> So, I have the rules as follows:
>
>
> # match 'n' skip already marked traffic
> /sbin/iptables -t mangle -A PREROUTING -m connmark --mark 0x8000/0x8000 \
> 	-j CONNMARK --restore-mark --mask 0xffff
> /sbin/iptables -t mangle -A PREROUTING -m connmark --mark 0x8000/0x8000 \
> 	-j RETURN
> #
> # Assign different marks to different traffic:
> # 192.168.1.218/32 is my server IP address and
> # the traffic refers to Ingress (Egress will be similar)
>
> # create a class :3280x
> /sbin/iptables -t mangle -N class_3280x
> /sbin/iptables -t mangle -A PREROUTING -i eth1 \
> 	-d ! 192.168.1.218/255.255.255.255 -j class_3280x
> # mark the traffic matchin this class and return
> # class :32801	(section 1)
> /sbin/iptables -t mangle -A class_3280x -j CONNMARK --set-mark
> 0x8021/0xFFFF /sbin/iptables -t mangle -A class_3280x -j RETURN
>
> # create a class: 6550x
> /sbin/iptables -t mangle -N class_6550x
> /sbin/iptables -t mangle -A PREROUTING -i eth1 \
> 	-d 192.168.1.218/255.255.255.255 -j class_6550x
> # mark the traffic matching this class and return
> # class 65501 (section 2)
> /sbin/iptables -t mangle -A class_6550x -j CONNMARK --set-mark
> 0xFFDD/0xFFFF /sbin/iptables -t mangle -A class_6550x -j RETURN
>
> # iprtoute2 tc stuff to regulate the traffic
> #
> # Ingress
> /sbin/tc qdisc add dev eth1 handle ffff: ingress
>
> # class :32801 (0x8021)
> /sbin/tc filter add dev eth1 parent ffff: protocol ip prio 10 \
>         u32 match mark 0x8021 0xffff \
>         police rate 512kbit burst 512kbit mtu 1500 drop flowid :1
>
> # class :65501  (0xFFDD)
> /sbin/tc filter add dev eth1 parent ffff: protocol ip prio 10 \
>         u32 match mark 0xFFDD 0xffff \
>         police rate 256kbit burst 256kbit mtu 1500 drop flowid :1
>
>
> Well, I generate some traffic on the network of eth1 to match the above
> rules. I can see first packets (SYN if the rules are applied before the
> connections establishes) matching the iptables rules for each class and
> then the flow is matched by the first 2 rules only (which is what I
> intended to do - skip the rules if traffic already marked). Now, I can also
> see the marked connections using
>
> cat /proc/net/ip_conntrack
>
> which is fine.
>
>
> When this setup works the traffic matches the tc rules, no probs, and it is
> assigned the bandwidth I want. When it doesn't work... the traffic matches
> the iptables rules in the same way when it's working, I can still track the
> connections marks in /proc/net/ip_conntrack, but the tc rules are never
> applied (I can see the stats for tc intercepting the traffic but not
> matching the marks, it's like they are not there).
>
> Some will say that this is related with iproute2 tools, but I have came
> across with this situation: instead of CONNTRACK marking, I do normal MARK
> and save the marks with CONNTRACK (just before RETURN), which means that in
> sections 1 & 2 identified above (for iptables rules) I rewrite the rules
> like this:
>
> /sbin/iptables -t mangle -A class_3280x -j MARK --set-mark 0x8021
> /sbin/iptables -t mangle -A class_3280x -j CONNMARK --save-mark
> /sbin/iptables -t mangle -A class_3280x -j RETURN
>
> respectively,
>
> /sbin/iptables -t mangle -A class_6550x -j MARK --set-mark 0xFFDD
> /sbin/iptables -t mangle -A class_3280x -j CONNMARK --save-mark
> /sbin/iptables -t mangle -A class_6550x -j RETURN
>
> This works always for me, I have try it dozens of time with no problem,
> whilst the --set-mark in CONNTRACK works sometime 1 out of 10 tries or
> never. It looks that setting the mark per connection is not that accurate
> and I need to remark every packet if unmarked? Am I doing something wrong?
>
> I'm using custom built kernel 2.6.11.6 and iptables 1.3.1. This rather
> strange behaviour occurs (for at least I could tell) when restarting the
> linux box and reaplying the rules or just deleting the rules and reaplying
> them again or inserting new matching rules.
>
> Any help will be much appreciated,
> Adrian
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (GNU/Linux)

iD8DBQFCXmLpeUXaRUbEOnARAqZfAJ0eqsVixHtTbsBLBHdWOrR9JZDcgACeJfnX
OHtD1Y6zBtqyofhkNzy0100=
=rqzh
-----END PGP SIGNATURE-----

Re: I need help with "CONNMARK --set-mark"

[Jason Opperisano]

On Thu, Apr 14, 2005 at 01:32:33PM +0100, Adrian Turcu wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
> 
> I'm just trying to revive this as there was no answer till now. Maybe I'm 
> lucky this time :)

i didn't respond because i couldn't really decipher what your actual
question was.  if your question is:

why does "-j MARK --set-mark" seem to work while "-j CONNMARK
--set-mark" seem to not work?  the best explanation i've seen posted
here was:

  http://marc.theaimsgroup.com/?l=netfilter&m=110894240806358&w=2

if that's not your question--well that would explain why i didn't
respond the first time.

-j

--
"Stewie: My God, I'm to entrust my life to a turtle? Nature's "D"
 student!"
        --Family Guy

Re: I need help with "CONNMARK --set-mark"

[Filip Sneppe]

Hi,

On 4/14/05, Jason Opperisano <opie@817west.com> wrote:
> On Thu, Apr 14, 2005 at 01:32:33PM +0100, Adrian Turcu wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > I'm just trying to revive this as there was no answer till now. Maybe I'm
> > lucky this time :)
> 
I haven't followed this in great detail either, but I do know that
some people reported problems with the firewall MARK in combination
with tc filter ... fwmark on the LARTC mailing list. I do remember someone
explicitly using a u32 mark match too.

I don't know what the details were, but could you give a different kernel
a try ? Sorry, I thought I kept the exact mail thread in my mailbox, but I
must have deleted it... You may want to check the LARTC archives for more 
info. 

So try a different kernel first, if the problem persists and you're convinced
that this is a bug, you may want to take this to the -devel mailing list, where
the author of the CONNMARK patch also hangs out.

Regards,
Filip

Re: I need help with "CONNMARK --set-mark"

[Adrian Turcu]

Thanks for your prompt answer Jason.

On Thu 14 Apr 2005 13:45, Jason Opperisano wrote:
> On Thu, Apr 14, 2005 at 01:32:33PM +0100, Adrian Turcu wrote:
> > -----BEGIN PGP SIGNED MESSAGE-----
> > Hash: SHA1
> >
> > I'm just trying to revive this as there was no answer till now. Maybe I'm
> > lucky this time :)
>
> i didn't respond because i couldn't really decipher what your actual
> question was.  if your question is:

It seems that I have to remember to switch off the message signing - company 
policy: signed messages only - not to interfere with sendig to the list.

I apologise to you all.

>
> why does "-j MARK --set-mark" seem to work while "-j CONNMARK
> --set-mark" seem to not work?  the best explanation i've seen posted
> here was:
>
>   http://marc.theaimsgroup.com/?l=netfilter&m=110894240806358&w=2
>
> if that's not your question--well that would explain why i didn't
> respond the first time.
>
> -j
>

Well, bottom line, that was my question, kind of, and that link you sent, it 
gives a very good explanation to the matter. I could not see this documented  
anywhere, I mean how CONNMARK behaves with its various options, maybe I did 
not read the right stuff. But anyway, great job! I have to stick with both of 
them, CONNMARK and the good old MARK.

Thanks again,
Adrian

Re: I need help with "CONNMARK --set-mark"

[Adrian Turcu]

Hi Filip,

On Thu 14 Apr 2005 14:09, Filip Sneppe wrote:
>
> I haven't followed this in great detail either, but I do know that
> some people reported problems with the firewall MARK in combination
> with tc filter ... fwmark on the LARTC mailing list. I do remember someone
> explicitly using a u32 mark match too.
>
> I don't know what the details were, but could you give a different kernel
> a try ? Sorry, I thought I kept the exact mail thread in my mailbox, but I
> must have deleted it... You may want to check the LARTC archives for more
> info.
>
> So try a different kernel first, if the problem persists and you're
> convinced that this is a bug, you may want to take this to the -devel
> mailing list, where the author of the CONNMARK patch also hangs out.
>
> Regards,
> Filip

I have tried with 2 different kernels 2.6.10 and 2.6.11.6 and 2 different 
iptables 1.3.0 and 1.3.1.  The behaviour is the same, most of the time the 
CONNMARK does not work.
I posted the 2.6.11.6 kernel with 1.3.1 iptables, 'cause I intend to use the 
latest stable versions for both.

I cannot say if this is a bug or not after reading through the message posted 
by Jason. Although, if you go to the link posted by Jason, it looks like the 
options of setting/saving the marks that CONNMARK has, they don't work as 
most people will expect to (set mark to the connection and _save_ it 
afterward inside the connetion, not resetting to zero and save it).

I can post the original message to the devel list, maybe I'll get a different 
twist from the author (if still there)


Regards,
Adrian