Mark traffic on one machine, match on another machine?

Mark traffic on one machine, match on another machine?

[Jack Bates]

Is there a way to mark traffic on one machine (proxy server) and match 
the mark on another machine (router)?

I want to shape web traffic differently depending, for example, on the 
requested domain. Our proxy server already knows about criteria like the 
requested domain, and our router already does traffic shaping, but the 
proxy and the router are on two separate machines. Is there a way to 
mark traffic on one machine and match the mark on another machine? so I 
can classify traffic on the proxy server and shape it on the router?

Re: Mark traffic on one machine, match on another machine?

[Steven Kath]

> Is there a way to mark traffic on one machine and match the 
> mark on another machine? so I can classify traffic on the 
> proxy server and shape it on the router?

This question is a good example of the rationale for the 
TOS/DSCP header on IPv4 packets. netfilter/iptables are 
quite capable of matching and manipulating the DSCP field, 
as are some proxy servers. 

Re: Mark traffic on one machine, match on another machine?

[Giles Coochey]

On 28-11-2012 05:25, Steven Kath wrote:
>> Is there a way to mark traffic on one machine and match the
>> mark on another machine? so I can classify traffic on the
>> proxy server and shape it on the router?
>
> This question is a good example of the rationale for the
> TOS/DSCP header on IPv4 packets. netfilter/iptables are
> quite capable of matching and manipulating the DSCP field,
> as are some proxy servers.
> --

+1 to above.

Alternatively, you could route the packet from the proxy to a 
different, secondary IP on the router. The router could then shape all 
the traffic that arrives on the secondary IP. You can achieve the 
secondary IP with sub-interfaces, secondary IP in the same subnet, or 
separate VLAN interfaces

I don't think there is any net-filter tagging, in the way that you can 
tag packets in rules while it passes internally through the proxy, that 
would be visible externally to the router.

Re: Mark traffic on one machine, match on another machine?

[Jack Bates]

On 28/11/12 04:54 AM, Giles Coochey wrote:
> On 28-11-2012 05:25, Steven Kath wrote:
>>> Is there a way to mark traffic on one machine and match the
>>> mark on another machine? so I can classify traffic on the
>>> proxy server and shape it on the router?
>>
>> This question is a good example of the rationale for the
>> TOS/DSCP header on IPv4 packets. netfilter/iptables are
>> quite capable of matching and manipulating the DSCP field,
>> as are some proxy servers.
>> --
>
> +1 to above.
>
> Alternatively, you could route the packet from the proxy to a different,
> secondary IP on the router. The router could then shape all the traffic
> that arrives on the secondary IP. You can achieve the secondary IP with
> sub-interfaces, secondary IP in the same subnet, or separate VLAN
> interfaces
>
> I don't think there is any net-filter tagging, in the way that you can
> tag packets in rules while it passes internally through the proxy, that
> would be visible externally to the router.

Cool, thanks a lot for this advice Steven and Giles, our proxy server is 
Apache Traffic Server, so I started work on a simple "remap" plugin to 
set TOS/DSCP field: http://nottheoilrig.com/trafficserver/201211300/tos.cc

It should enable something like the following, in the Traffic Server 
remap.config:

   map http://gmail.com @plugin=tos.so @pparam=3
   map http://facebook.com @plugin=tos.so @pparam=7

But what about response traffic? Is there a way to copy the TOS/DSCP 
field to the response from the origin server?

Re: Mark traffic on one machine, match on another machine?

[Eliezer Croitoru]

You can do a connection marking\remarking based on the TOS of one packet.
This way you need to set it only on even one packet of the connection.

Regards,
Eliezer

On 11/30/2012 7:41 AM, Jack Bates wrote:
> Cool, thanks a lot for this advice Steven and Giles, our proxy server is
> Apache Traffic Server, so I started work on a simple "remap" plugin to
> set TOS/DSCP field: http://nottheoilrig.com/trafficserver/201211300/tos.cc
>
> It should enable something like the following, in the Traffic Server
> remap.config:
>
>    map http://gmail.com @plugin=tos.so @pparam=3
>    map http://facebook.com @plugin=tos.so @pparam=7
>
> But what about response traffic? Is there a way to copy the TOS/DSCP
> field to the response from the origin server?

Re: Mark traffic on one machine, match on another machine?

[Jack Bates]

Thank you Eliezer, how do I shape traffic differently depending on the 
connection mark?

Currently I use the ingress qdisc plus ifb0 plus the tbf qdisc to limit 
the *sum* of our upstream plus downstream traffic to < 1.5mbit (to "own" 
the queue). Something like:

    1: ifconfig ifb0 up
    2:
    3: insmod sch_ingress
    4: tc qdisc add dev eth0.2 ingress
    5:
    6: insmod cls_u32
    7: insmod act_mirred
    8: tc filter add dev eth0.2 root protocol ip u32 match u32 0 0 
action mirred egress redirect dev ifb0
    9: tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0 
0 action mirred egress redirect dev ifb0
   10:
   11: insmod sch_tbf
   12: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k 
latency 70ms

Then I use the prio qdisc to prioritize traffic:

   14: insmod sch_prio
   15: tc qdisc add dev ifb0 parent 1: handle 2 prio

I can imagine a couple ways of classifying traffic from our proxy server 
based on the TOS/DSCP field, and also how to set the connection mark 
based on this field. But how do I classify and shape response traffic 
from the origin server based on the connection mark?

On 29/11/12 10:27 PM, Eliezer Croitoru wrote:
> You can do a connection marking\remarking based on the TOS of one packet.
> This way you need to set it only on even one packet of the connection.
>
> Regards,
> Eliezer
>
> On 11/30/2012 7:41 AM, Jack Bates wrote:
>> Cool, thanks a lot for this advice Steven and Giles, our proxy server is
>> Apache Traffic Server, so I started work on a simple "remap" plugin to
>> set TOS/DSCP field:
>> http://nottheoilrig.com/trafficserver/201211300/tos.cc
>>
>> It should enable something like the following, in the Traffic Server
>> remap.config:
>>
>>    map http://gmail.com @plugin=tos.so @pparam=3
>>    map http://facebook.com @plugin=tos.so @pparam=7
>>
>> But what about response traffic? Is there a way to copy the TOS/DSCP
>> field to the response from the origin server?

Re: Mark traffic on one machine, match on another machine?

[Eliezer Croitoru]

You use iptables mark + restore mark based on connection tracking.
you can mark the TOS on the outgoing postrouting table.
you can take a look at the iptabes man pages:
http://ipset.netfilter.org/iptables.man.html
which has --restore-mark  exaple.

Eliezer

On 12/3/2012 10:43 AM, Jack Bates wrote:
> I can imagine a couple ways of classifying traffic from our proxy server
> based on the TOS/DSCP field, and also how to set the connection mark
> based on this field. But how do I classify and shape response traffic
> from the origin server based on the connection mark?

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
sip:ngtech@sip2sip.info
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il

Re: Mark traffic on one machine, match on another machine?

[Jack Bates]

Thanks a lot for your help, how can I evaluate --restore-mark before I 
classify and shape response traffic from the origin server?

I think you mean something like:

   # Copy ctmart to nfmark (e.g. 1, 2)
   iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK --restore-mark

   # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class 2:2
   tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw flowid 
2:1 action mirred egress redirect dev ifb0
   tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw flowid 
2:3 action mirred egress redirect dev ifb0
   tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0 0 
flowid 2:2 action mirred egress redirect dev ifb0

Just how can I get --restore-mark to evaluate before tc filter?

Another way I can imagine is with the CLASSIFY target:

   # Send unmarked traffic to class 2:2
   iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1 -j 
CLASSIFY 2:1
   iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2 -j 
CLASSIFY 2:3
   iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2

But I have the same challenge, how can I evaluate the CLASSIFY target 
before I shape traffic?

Or is there another way to classify and shape response traffic from the 
origin server based on the TOS/DSCP field of the request?

On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
> You use iptables mark + restore mark based on connection tracking.
> you can mark the TOS on the outgoing postrouting table.
> you can take a look at the iptabes man pages:
> http://ipset.netfilter.org/iptables.man.html
> which has --restore-mark  exaple.
>
> Eliezer
>
> On 12/3/2012 10:43 AM, Jack Bates wrote:
>> I can imagine a couple ways of classifying traffic from our proxy server
>> based on the TOS/DSCP field, and also how to set the connection mark
>> based on this field. But how do I classify and shape response traffic
>> from the origin server based on the connection mark?

Re: Mark traffic on one machine, match on another machine?

[Anatoly Muliarski]

Hi Jack,

--restore-mark should be used for existing connections and to mark new
ones you may use something like that:

iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m conntrack
--cstate NEW,RELATED -j MARK --set-mark 1

The main idea consists in marking packets on the physical input
interface and shaping them on ifb0( where they arrive already marked
).
iptables' packet marks exist only in memory of one computer, TOS/DSCP
may be used for transmitting a map of them to another one.
BTW, use --restore-mark on the output interface of your shaper too.

2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
> Thanks a lot for your help, how can I evaluate --restore-mark before I
> classify and shape response traffic from the origin server?
>
> I think you mean something like:
>
>    # Copy ctmart to nfmark (e.g. 1, 2)
>    iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK --restore-mark
>
>    # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class 2:2
>    tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw flowid
> 2:1 action mirred egress redirect dev ifb0
>    tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw flowid
> 2:3 action mirred egress redirect dev ifb0
>    tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0 0
> flowid 2:2 action mirred egress redirect dev ifb0
>
> Just how can I get --restore-mark to evaluate before tc filter?
>
> Another way I can imagine is with the CLASSIFY target:
>
>    # Send unmarked traffic to class 2:2
>    iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1 -j
> CLASSIFY 2:1
>    iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2 -j
> CLASSIFY 2:3
>    iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>
> But I have the same challenge, how can I evaluate the CLASSIFY target
> before I shape traffic?
>
> Or is there another way to classify and shape response traffic from the
> origin server based on the TOS/DSCP field of the request?
>
> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>> You use iptables mark + restore mark based on connection tracking.
>> you can mark the TOS on the outgoing postrouting table.
>> you can take a look at the iptabes man pages:
>> http://ipset.netfilter.org/iptables.man.html
>> which has --restore-mark  exaple.
>>
>> Eliezer
>>
>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>> I can imagine a couple ways of classifying traffic from our proxy server
>>> based on the TOS/DSCP field, and also how to set the connection mark
>>> based on this field. But how do I classify and shape response traffic
>>> from the origin server based on the connection mark?
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>


-- 
Best regards
Anatoly Muliarski

Re: Mark traffic on one machine, match on another machine?

[Eliezer Croitoru]

Thanks Anatoly,

My idea was that based on a mark he will jump the packets to another 
table which HE will mark TOS/DSCP.

Eliezer

On 12/5/2012 4:39 AM, Anatoly Muliarski wrote:
> Hi Jack,
>
> --restore-mark should be used for existing connections and to mark new
> ones you may use something like that:
>
> iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m conntrack
> --cstate NEW,RELATED -j MARK --set-mark 1
>
> The main idea consists in marking packets on the physical input
> interface and shaping them on ifb0( where they arrive already marked
> ).
> iptables' packet marks exist only in memory of one computer, TOS/DSCP
> may be used for transmitting a map of them to another one.
> BTW, use --restore-mark on the output interface of your shaper too.
>
> 2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
>> Thanks a lot for your help, how can I evaluate --restore-mark before I
>> classify and shape response traffic from the origin server?
>>
>> I think you mean something like:
>>
>>     # Copy ctmart to nfmark (e.g. 1, 2)
>>     iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK --restore-mark
>>
>>     # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class 2:2
>>     tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw flowid
>> 2:1 action mirred egress redirect dev ifb0
>>     tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw flowid
>> 2:3 action mirred egress redirect dev ifb0
>>     tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0 0
>> flowid 2:2 action mirred egress redirect dev ifb0
>>
>> Just how can I get --restore-mark to evaluate before tc filter?
>>
>> Another way I can imagine is with the CLASSIFY target:
>>
>>     # Send unmarked traffic to class 2:2
>>     iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1 -j
>> CLASSIFY 2:1
>>     iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2 -j
>> CLASSIFY 2:3
>>     iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>>
>> But I have the same challenge, how can I evaluate the CLASSIFY target
>> before I shape traffic?
>>
>> Or is there another way to classify and shape response traffic from the
>> origin server based on the TOS/DSCP field of the request?
>>
>> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>>> You use iptables mark + restore mark based on connection tracking.
>>> you can mark the TOS on the outgoing postrouting table.
>>> you can take a look at the iptabes man pages:
>>> http://ipset.netfilter.org/iptables.man.html
>>> which has --restore-mark  exaple.
>>>
>>> Eliezer
>>>
>>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>>> I can imagine a couple ways of classifying traffic from our proxy server
>>>> based on the TOS/DSCP field, and also how to set the connection mark
>>>> based on this field. But how do I classify and shape response traffic
>>>> from the origin server based on the connection mark?
>> --
>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>
>

-- 
Eliezer Croitoru
https://www1.ngtech.co.il
sip:ngtech@sip2sip.info
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il

Re: Mark traffic on one machine, match on another machine?

[Jack Bates]

Thanks for this help, I think the marks are getting saved/restored 
successfully, and I can successfully classify request traffic from the 
proxy server based on the mark (statistics on the eth0.2 classes are 
incrementing) but response traffic from the origin server isn't getting 
classified yet (statistics on the ifb0 classes remain zero).

Here is my complete work in progress script. How can I investigate why 
response traffic isn't getting classified? Are there any more details 
that would help explain why not?

How can I classify traffic based on the restored mark? or is there 
another way to classify response traffic from the origin server based on 
the TOS/DSCP field of the request?

    1: # Save/restore connection mark based on TOS field
    2: iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
    3: iptables -A PREROUTING -t mangle -m tos --tos 12 -j MARK --set-mark 1
    4: iptables -A PREROUTING -t mangle -m tos --tos 28 -j MARK --set-mark 2
    5: iptables -A PREROUTING -t mangle -j CONNMARK --save-mark
    6:
    7: ifconfig ifb0 up
    8:
    9: insmod sch_ingress
   10: tc qdisc add dev eth0.2 ingress
   11:
   12: # Classify response traffic from origin server based on restored mark
   13: insmod cls_fw
   14: insmod act_mirred
   15: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1 
fw flowid 2:1 action mirred egress redirect dev ifb0
   16: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 2 
fw flowid 2:3 action mirred egress redirect dev ifb0
   17:
   18: # Send unmarked traffic to class 2:2
   19: insmod cls_u32
   20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 u32 
match u32 0 0 flowid 2:2 action mirred egress redirect dev ifb0
   21:
   22: # Limit up and downstream to 1mbit, to "own" the queue
   23: insmod sch_tbf
   24: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k 
latency 70ms
   25: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k 
latency 70ms
   26:
   27: # High, low, and "other" priority
   28: insmod sch_prio
   29: tc qdisc add dev eth0.2 parent 1: handle 2 prio
   30: tc qdisc add dev ifb0 parent 1: handle 2 prio
   31:
   32: # Classify request traffic from proxy server based on mark
   33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw 
flowid :1
   34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw 
flowid :3

On 05/12/12 01:12 AM, Eliezer Croitoru wrote:
> Thanks Anatoly,
>
> My idea was that based on a mark he will jump the packets to another
> table which HE will mark TOS/DSCP.
>
> Eliezer
>
> On 12/5/2012 4:39 AM, Anatoly Muliarski wrote:
>> Hi Jack,
>>
>> --restore-mark should be used for existing connections and to mark new
>> ones you may use something like that:
>>
>> iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m conntrack
>> --cstate NEW,RELATED -j MARK --set-mark 1
>>
>> The main idea consists in marking packets on the physical input
>> interface and shaping them on ifb0( where they arrive already marked
>> ).
>> iptables' packet marks exist only in memory of one computer, TOS/DSCP
>> may be used for transmitting a map of them to another one.
>> BTW, use --restore-mark on the output interface of your shaper too.
>>
>> 2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
>>> Thanks a lot for your help, how can I evaluate --restore-mark before I
>>> classify and shape response traffic from the origin server?
>>>
>>> I think you mean something like:
>>>
>>>     # Copy ctmart to nfmark (e.g. 1, 2)
>>>     iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK
>>> --restore-mark
>>>
>>>     # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class 2:2
>>>     tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw flowid
>>> 2:1 action mirred egress redirect dev ifb0
>>>     tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw flowid
>>> 2:3 action mirred egress redirect dev ifb0
>>>     tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0 0
>>> flowid 2:2 action mirred egress redirect dev ifb0
>>>
>>> Just how can I get --restore-mark to evaluate before tc filter?
>>>
>>> Another way I can imagine is with the CLASSIFY target:
>>>
>>>     # Send unmarked traffic to class 2:2
>>>     iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1 -j
>>> CLASSIFY 2:1
>>>     iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2 -j
>>> CLASSIFY 2:3
>>>     iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>>>
>>> But I have the same challenge, how can I evaluate the CLASSIFY target
>>> before I shape traffic?
>>>
>>> Or is there another way to classify and shape response traffic from the
>>> origin server based on the TOS/DSCP field of the request?
>>>
>>> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>>>> You use iptables mark + restore mark based on connection tracking.
>>>> you can mark the TOS on the outgoing postrouting table.
>>>> you can take a look at the iptabes man pages:
>>>> http://ipset.netfilter.org/iptables.man.html
>>>> which has --restore-mark  exaple.
>>>>
>>>> Eliezer
>>>>
>>>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>>>> I can imagine a couple ways of classifying traffic from our proxy
>>>>> server
>>>>> based on the TOS/DSCP field, and also how to set the connection mark
>>>>> based on this field. But how do I classify and shape response traffic
>>>>> from the origin server based on the connection mark?
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>
>>
>

Re: Mark traffic on one machine, match on another machine?

[Anatoly Muliarski]

IMHO, there is a bit of mess in your scheme.
Let's clarify it :)

If my assumptions are right then:
1. eth0.2 is the external shaper interface.
2. ifb0 is using for shaping incoming traffic on this interface.

To set marks for the traffic you should use something like these:

iptables -t manlge -N NEW_CONN
iptables -t mangle -A NEW_CONN -m tos --tos 12 -j MARK --set-mark 1
iptables -t mangle -A NEW_CONN -m tos --tos 28 -j MARK --set-mark 2
# to mark all the unmatched traffic and treat it separately in your shaper
iptables -t mangle -A NEW_CONN -m mark --mark 0  -j MARK --set-mark 3
iptables -t mangle -A NEW_CONN  -j CONNMARK --save-mark

iptables -t mangle -A PREROUTING -m conntrack --cstate INVALID -j DROP
iptables -t mangle -A PREROUTING -m conntrack --cstate ESTABLISHED -j
CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m conntrack --cstate NEW,RELATED -g NEW_CONN


2012/12/5, Jack Bates <uo4zau@nottheoilrig.com>:
> Thanks for this help, I think the marks are getting saved/restored
> successfully, and I can successfully classify request traffic from the
> proxy server based on the mark (statistics on the eth0.2 classes are
> incrementing) but response traffic from the origin server isn't getting
> classified yet (statistics on the ifb0 classes remain zero).
>
> Here is my complete work in progress script. How can I investigate why
> response traffic isn't getting classified? Are there any more details
> that would help explain why not?
>
> How can I classify traffic based on the restored mark? or is there
> another way to classify response traffic from the origin server based on
> the TOS/DSCP field of the request?
>
>     1: # Save/restore connection mark based on TOS field
>     2: iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
>     3: iptables -A PREROUTING -t mangle -m tos --tos 12 -j MARK --set-mark
> 1
>     4: iptables -A PREROUTING -t mangle -m tos --tos 28 -j MARK --set-mark
> 2
>     5: iptables -A PREROUTING -t mangle -j CONNMARK --save-mark
>     6:
>     7: ifconfig ifb0 up
>     8:
>     9: insmod sch_ingress
>    10: tc qdisc add dev eth0.2 ingress
>    11:
>    12: # Classify response traffic from origin server based on restored
> mark
>    13: insmod cls_fw
>    14: insmod act_mirred
>    15: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1
> fw flowid 2:1 action mirred egress redirect dev ifb0
>    16: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 2
> fw flowid 2:3 action mirred egress redirect dev ifb0
>    17:
>    18: # Send unmarked traffic to class 2:2
>    19: insmod cls_u32
>    20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 u32
> match u32 0 0 flowid 2:2 action mirred egress redirect dev ifb0
>    21:
>    22: # Limit up and downstream to 1mbit, to "own" the queue
>    23: insmod sch_tbf
>    24: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
> latency 70ms
>    25: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
> latency 70ms
>    26:
>    27: # High, low, and "other" priority
>    28: insmod sch_prio
>    29: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>    30: tc qdisc add dev ifb0 parent 1: handle 2 prio
>    31:
>    32: # Classify request traffic from proxy server based on mark
>    33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw
> flowid :1
>    34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw
> flowid :3
>
> On 05/12/12 01:12 AM, Eliezer Croitoru wrote:
>> Thanks Anatoly,
>>
>> My idea was that based on a mark he will jump the packets to another
>> table which HE will mark TOS/DSCP.
>>
>> Eliezer
>>
>> On 12/5/2012 4:39 AM, Anatoly Muliarski wrote:
>>> Hi Jack,
>>>
>>> --restore-mark should be used for existing connections and to mark new
>>> ones you may use something like that:
>>>
>>> iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m conntrack
>>> --cstate NEW,RELATED -j MARK --set-mark 1
>>>
>>> The main idea consists in marking packets on the physical input
>>> interface and shaping them on ifb0( where they arrive already marked
>>> ).
>>> iptables' packet marks exist only in memory of one computer, TOS/DSCP
>>> may be used for transmitting a map of them to another one.
>>> BTW, use --restore-mark on the output interface of your shaper too.
>>>
>>> 2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
>>>> Thanks a lot for your help, how can I evaluate --restore-mark before I
>>>> classify and shape response traffic from the origin server?
>>>>
>>>> I think you mean something like:
>>>>
>>>>     # Copy ctmart to nfmark (e.g. 1, 2)
>>>>     iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK
>>>> --restore-mark
>>>>
>>>>     # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class
>>>> 2:2
>>>>     tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw
>>>> flowid
>>>> 2:1 action mirred egress redirect dev ifb0
>>>>     tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw
>>>> flowid
>>>> 2:3 action mirred egress redirect dev ifb0
>>>>     tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0 0
>>>> flowid 2:2 action mirred egress redirect dev ifb0
>>>>
>>>> Just how can I get --restore-mark to evaluate before tc filter?
>>>>
>>>> Another way I can imagine is with the CLASSIFY target:
>>>>
>>>>     # Send unmarked traffic to class 2:2
>>>>     iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1 -j
>>>> CLASSIFY 2:1
>>>>     iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2 -j
>>>> CLASSIFY 2:3
>>>>     iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>>>>
>>>> But I have the same challenge, how can I evaluate the CLASSIFY target
>>>> before I shape traffic?
>>>>
>>>> Or is there another way to classify and shape response traffic from the
>>>> origin server based on the TOS/DSCP field of the request?
>>>>
>>>> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>>>>> You use iptables mark + restore mark based on connection tracking.
>>>>> you can mark the TOS on the outgoing postrouting table.
>>>>> you can take a look at the iptabes man pages:
>>>>> http://ipset.netfilter.org/iptables.man.html
>>>>> which has --restore-mark  exaple.
>>>>>
>>>>> Eliezer
>>>>>
>>>>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>>>>> I can imagine a couple ways of classifying traffic from our proxy
>>>>>> server
>>>>>> based on the TOS/DSCP field, and also how to set the connection mark
>>>>>> based on this field. But how do I classify and shape response traffic
>>>>>> from the origin server based on the connection mark?
>>>> --
>>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>>> the body of a message to majordomo@vger.kernel.org
>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>
>>>
>>>
>>
>


-- 
Best regards
Anatoly Muliarski

Re: Mark traffic on one machine, match on another machine?

[Jack Bates]

Thank you for this clarification, and your assumptions are correct: I'm 
trying to prioritize incoming and outgoing traffic on eth0.2, ifb0 is 
for shaping the incoming traffic.

But I'm still failing to filter incoming traffic based on the restored 
mark: statistics on the ifb0 qdiscs and classes remain zero. Filtering 
outgoing traffic is working as expected, statistics on the eth0.2 
classes are incrementing as expected, which I think means that the 
PREROUTING --restore-mark target is working?

Do you have any other suggestions how to investigate why incoming 
traffic isn't getting filtered? or is there another way to classify 
response traffic from the origin server based on the TOS/DSCP field of 
the request from the proxy server? Are there any more details that would 
help explain the problem?

Here is the clarified script:

    1: iptables -N NEW_CONN -t mangle
    2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark 1
    3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark 2
    4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK --set-mark 3
    5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
    6:
    7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID 
-j DROP
    8: iptables -A PREROUTING -t mangle -m conntrack --ctstate 
ESTABLISHED -j CONNMARK --restore-mark
    9: iptables -A PREROUTING -t mangle -m conntrack --ctstate 
NEW,RELATED -j NEW_CONN
   10:
   11: ifconfig ifb0 up
   12:
   13: insmod sch_ingress
   14: tc qdisc add dev eth0.2 ingress
   15:
   16: # Classify response traffic from origin server based on restored mark
   17: insmod cls_fw
   18: insmod act_mirred
   19: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1 
fw flowid 2:1 action mirred egress redirect dev ifb0
   20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 2 
fw flowid 2:3 action mirred egress redirect dev ifb0
   21: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 3 
fw flowid 2:2 action mirred egress redirect dev ifb0
   22:
   23: # Limit up and downstream to 1mbit, to "own" the queue
   24: insmod sch_tbf
   25: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k 
latency 70ms
   26: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k 
latency 70ms
   27:
   28: # High, low, and "other" priority
   29: insmod sch_prio
   30: tc qdisc add dev eth0.2 parent 1: handle 2 prio
   31: tc qdisc add dev ifb0 parent 1: handle 2 prio
   32:
   33: # Classify request traffic from proxy server based on mark
   34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw 
flowid :1
   35: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw 
flowid :3
   36: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 3 fw 
flowid :2

On 05/12/12 08:18 PM, Anatoly Muliarski wrote:
> IMHO, there is a bit of mess in your scheme.
> Let's clarify it :)
>
> If my assumptions are right then:
> 1. eth0.2 is the external shaper interface.
> 2. ifb0 is using for shaping incoming traffic on this interface.
>
> To set marks for the traffic you should use something like these:
>
> iptables -t manlge -N NEW_CONN
> iptables -t mangle -A NEW_CONN -m tos --tos 12 -j MARK --set-mark 1
> iptables -t mangle -A NEW_CONN -m tos --tos 28 -j MARK --set-mark 2
> # to mark all the unmatched traffic and treat it separately in your shaper
> iptables -t mangle -A NEW_CONN -m mark --mark 0  -j MARK --set-mark 3
> iptables -t mangle -A NEW_CONN  -j CONNMARK --save-mark
>
> iptables -t mangle -A PREROUTING -m conntrack --cstate INVALID -j DROP
> iptables -t mangle -A PREROUTING -m conntrack --cstate ESTABLISHED -j
> CONNMARK --restore-mark
> iptables -t mangle -A PREROUTING -m conntrack --cstate NEW,RELATED -g NEW_CONN
>
>
> 2012/12/5, Jack Bates <uo4zau@nottheoilrig.com>:
>> Thanks for this help, I think the marks are getting saved/restored
>> successfully, and I can successfully classify request traffic from the
>> proxy server based on the mark (statistics on the eth0.2 classes are
>> incrementing) but response traffic from the origin server isn't getting
>> classified yet (statistics on the ifb0 classes remain zero).
>>
>> Here is my complete work in progress script. How can I investigate why
>> response traffic isn't getting classified? Are there any more details
>> that would help explain why not?
>>
>> How can I classify traffic based on the restored mark? or is there
>> another way to classify response traffic from the origin server based on
>> the TOS/DSCP field of the request?
>>
>>      1: # Save/restore connection mark based on TOS field
>>      2: iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
>>      3: iptables -A PREROUTING -t mangle -m tos --tos 12 -j MARK --set-mark
>> 1
>>      4: iptables -A PREROUTING -t mangle -m tos --tos 28 -j MARK --set-mark
>> 2
>>      5: iptables -A PREROUTING -t mangle -j CONNMARK --save-mark
>>      6:
>>      7: ifconfig ifb0 up
>>      8:
>>      9: insmod sch_ingress
>>     10: tc qdisc add dev eth0.2 ingress
>>     11:
>>     12: # Classify response traffic from origin server based on restored
>> mark
>>     13: insmod cls_fw
>>     14: insmod act_mirred
>>     15: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1
>> fw flowid 2:1 action mirred egress redirect dev ifb0
>>     16: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 2
>> fw flowid 2:3 action mirred egress redirect dev ifb0
>>     17:
>>     18: # Send unmarked traffic to class 2:2
>>     19: insmod cls_u32
>>     20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 u32
>> match u32 0 0 flowid 2:2 action mirred egress redirect dev ifb0
>>     21:
>>     22: # Limit up and downstream to 1mbit, to "own" the queue
>>     23: insmod sch_tbf
>>     24: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
>> latency 70ms
>>     25: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
>> latency 70ms
>>     26:
>>     27: # High, low, and "other" priority
>>     28: insmod sch_prio
>>     29: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>>     30: tc qdisc add dev ifb0 parent 1: handle 2 prio
>>     31:
>>     32: # Classify request traffic from proxy server based on mark
>>     33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw
>> flowid :1
>>     34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw
>> flowid :3
>>
>> On 05/12/12 01:12 AM, Eliezer Croitoru wrote:
>>> Thanks Anatoly,
>>>
>>> My idea was that based on a mark he will jump the packets to another
>>> table which HE will mark TOS/DSCP.
>>>
>>> Eliezer
>>>
>>> On 12/5/2012 4:39 AM, Anatoly Muliarski wrote:
>>>> Hi Jack,
>>>>
>>>> --restore-mark should be used for existing connections and to mark new
>>>> ones you may use something like that:
>>>>
>>>> iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m conntrack
>>>> --cstate NEW,RELATED -j MARK --set-mark 1
>>>>
>>>> The main idea consists in marking packets on the physical input
>>>> interface and shaping them on ifb0( where they arrive already marked
>>>> ).
>>>> iptables' packet marks exist only in memory of one computer, TOS/DSCP
>>>> may be used for transmitting a map of them to another one.
>>>> BTW, use --restore-mark on the output interface of your shaper too.
>>>>
>>>> 2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
>>>>> Thanks a lot for your help, how can I evaluate --restore-mark before I
>>>>> classify and shape response traffic from the origin server?
>>>>>
>>>>> I think you mean something like:
>>>>>
>>>>>      # Copy ctmart to nfmark (e.g. 1, 2)
>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK
>>>>> --restore-mark
>>>>>
>>>>>      # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class
>>>>> 2:2
>>>>>      tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw
>>>>> flowid
>>>>> 2:1 action mirred egress redirect dev ifb0
>>>>>      tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw
>>>>> flowid
>>>>> 2:3 action mirred egress redirect dev ifb0
>>>>>      tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0 0
>>>>> flowid 2:2 action mirred egress redirect dev ifb0
>>>>>
>>>>> Just how can I get --restore-mark to evaluate before tc filter?
>>>>>
>>>>> Another way I can imagine is with the CLASSIFY target:
>>>>>
>>>>>      # Send unmarked traffic to class 2:2
>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1 -j
>>>>> CLASSIFY 2:1
>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2 -j
>>>>> CLASSIFY 2:3
>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>>>>>
>>>>> But I have the same challenge, how can I evaluate the CLASSIFY target
>>>>> before I shape traffic?
>>>>>
>>>>> Or is there another way to classify and shape response traffic from the
>>>>> origin server based on the TOS/DSCP field of the request?
>>>>>
>>>>> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>>>>>> You use iptables mark + restore mark based on connection tracking.
>>>>>> you can mark the TOS on the outgoing postrouting table.
>>>>>> you can take a look at the iptabes man pages:
>>>>>> http://ipset.netfilter.org/iptables.man.html
>>>>>> which has --restore-mark  exaple.
>>>>>>
>>>>>> Eliezer
>>>>>>
>>>>>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>>>>>> I can imagine a couple ways of classifying traffic from our proxy
>>>>>>> server
>>>>>>> based on the TOS/DSCP field, and also how to set the connection mark
>>>>>>> based on this field. But how do I classify and shape response traffic
>>>>>>> from the origin server based on the connection mark?
>>>>> --
>>>>> To unsubscribe from this list: send the line "unsubscribe netfilter" in
>>>>> the body of a message to majordomo@vger.kernel.org
>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>
>>>>
>>>>
>>>
>>
>
>

Re: Mark traffic on one machine, match on another machine?

[Anatoly Muliarski]

IMHO, the problem lies in next - incoming packets on eth0.2 have no
mark and therefore they cannot be forwarded into ifb0 on the current
classification scheme. You should redirect all incoming packets( or
use non-mark redirection criteria ) into ifb0 and then classify and
shape them on ifb0 device.

To more clarify the way of a packet look at
http://en.wikipedia.org/wiki/File:Netfilter-packet-flow.svg

2012/12/10, Jack Bates <uo4zau@nottheoilrig.com>:
> Thank you for this clarification, and your assumptions are correct: I'm
> trying to prioritize incoming and outgoing traffic on eth0.2, ifb0 is
> for shaping the incoming traffic.
>
> But I'm still failing to filter incoming traffic based on the restored
> mark: statistics on the ifb0 qdiscs and classes remain zero. Filtering
> outgoing traffic is working as expected, statistics on the eth0.2
> classes are incrementing as expected, which I think means that the
> PREROUTING --restore-mark target is working?
>
> Do you have any other suggestions how to investigate why incoming
> traffic isn't getting filtered? or is there another way to classify
> response traffic from the origin server based on the TOS/DSCP field of
> the request from the proxy server? Are there any more details that would
> help explain the problem?
>
> Here is the clarified script:
>
>     1: iptables -N NEW_CONN -t mangle
>     2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark 1
>     3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark 2
>     4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK --set-mark 3
>     5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
>     6:
>     7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID
> -j DROP
>     8: iptables -A PREROUTING -t mangle -m conntrack --ctstate
> ESTABLISHED -j CONNMARK --restore-mark
>     9: iptables -A PREROUTING -t mangle -m conntrack --ctstate
> NEW,RELATED -j NEW_CONN
>    10:
>    11: ifconfig ifb0 up
>    12:
>    13: insmod sch_ingress
>    14: tc qdisc add dev eth0.2 ingress
>    15:
>    16: # Classify response traffic from origin server based on restored
> mark
>    17: insmod cls_fw
>    18: insmod act_mirred
>    19: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1
> fw flowid 2:1 action mirred egress redirect dev ifb0
>    20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 2
> fw flowid 2:3 action mirred egress redirect dev ifb0
>    21: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 3
> fw flowid 2:2 action mirred egress redirect dev ifb0
>    22:
>    23: # Limit up and downstream to 1mbit, to "own" the queue
>    24: insmod sch_tbf
>    25: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
> latency 70ms
>    26: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
> latency 70ms
>    27:
>    28: # High, low, and "other" priority
>    29: insmod sch_prio
>    30: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>    31: tc qdisc add dev ifb0 parent 1: handle 2 prio
>    32:
>    33: # Classify request traffic from proxy server based on mark
>    34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw
> flowid :1
>    35: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw
> flowid :3
>    36: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 3 fw
> flowid :2
>
> On 05/12/12 08:18 PM, Anatoly Muliarski wrote:
>> IMHO, there is a bit of mess in your scheme.
>> Let's clarify it :)
>>
>> If my assumptions are right then:
>> 1. eth0.2 is the external shaper interface.
>> 2. ifb0 is using for shaping incoming traffic on this interface.
>>
>> To set marks for the traffic you should use something like these:
>>
>> iptables -t manlge -N NEW_CONN
>> iptables -t mangle -A NEW_CONN -m tos --tos 12 -j MARK --set-mark 1
>> iptables -t mangle -A NEW_CONN -m tos --tos 28 -j MARK --set-mark 2
>> # to mark all the unmatched traffic and treat it separately in your
>> shaper
>> iptables -t mangle -A NEW_CONN -m mark --mark 0  -j MARK --set-mark 3
>> iptables -t mangle -A NEW_CONN  -j CONNMARK --save-mark
>>
>> iptables -t mangle -A PREROUTING -m conntrack --cstate INVALID -j DROP
>> iptables -t mangle -A PREROUTING -m conntrack --cstate ESTABLISHED -j
>> CONNMARK --restore-mark
>> iptables -t mangle -A PREROUTING -m conntrack --cstate NEW,RELATED -g
>> NEW_CONN
>>
>>
>> 2012/12/5, Jack Bates <uo4zau@nottheoilrig.com>:
>>> Thanks for this help, I think the marks are getting saved/restored
>>> successfully, and I can successfully classify request traffic from the
>>> proxy server based on the mark (statistics on the eth0.2 classes are
>>> incrementing) but response traffic from the origin server isn't getting
>>> classified yet (statistics on the ifb0 classes remain zero).
>>>
>>> Here is my complete work in progress script. How can I investigate why
>>> response traffic isn't getting classified? Are there any more details
>>> that would help explain why not?
>>>
>>> How can I classify traffic based on the restored mark? or is there
>>> another way to classify response traffic from the origin server based on
>>> the TOS/DSCP field of the request?
>>>
>>>      1: # Save/restore connection mark based on TOS field
>>>      2: iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
>>>      3: iptables -A PREROUTING -t mangle -m tos --tos 12 -j MARK
>>> --set-mark
>>> 1
>>>      4: iptables -A PREROUTING -t mangle -m tos --tos 28 -j MARK
>>> --set-mark
>>> 2
>>>      5: iptables -A PREROUTING -t mangle -j CONNMARK --save-mark
>>>      6:
>>>      7: ifconfig ifb0 up
>>>      8:
>>>      9: insmod sch_ingress
>>>     10: tc qdisc add dev eth0.2 ingress
>>>     11:
>>>     12: # Classify response traffic from origin server based on restored
>>> mark
>>>     13: insmod cls_fw
>>>     14: insmod act_mirred
>>>     15: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle
>>> 1
>>> fw flowid 2:1 action mirred egress redirect dev ifb0
>>>     16: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle
>>> 2
>>> fw flowid 2:3 action mirred egress redirect dev ifb0
>>>     17:
>>>     18: # Send unmarked traffic to class 2:2
>>>     19: insmod cls_u32
>>>     20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 u32
>>> match u32 0 0 flowid 2:2 action mirred egress redirect dev ifb0
>>>     21:
>>>     22: # Limit up and downstream to 1mbit, to "own" the queue
>>>     23: insmod sch_tbf
>>>     24: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
>>> latency 70ms
>>>     25: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
>>> latency 70ms
>>>     26:
>>>     27: # High, low, and "other" priority
>>>     28: insmod sch_prio
>>>     29: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>>>     30: tc qdisc add dev ifb0 parent 1: handle 2 prio
>>>     31:
>>>     32: # Classify request traffic from proxy server based on mark
>>>     33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1
>>> fw
>>> flowid :1
>>>     34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2
>>> fw
>>> flowid :3
>>>
>>> On 05/12/12 01:12 AM, Eliezer Croitoru wrote:
>>>> Thanks Anatoly,
>>>>
>>>> My idea was that based on a mark he will jump the packets to another
>>>> table which HE will mark TOS/DSCP.
>>>>
>>>> Eliezer
>>>>
>>>> On 12/5/2012 4:39 AM, Anatoly Muliarski wrote:
>>>>> Hi Jack,
>>>>>
>>>>> --restore-mark should be used for existing connections and to mark new
>>>>> ones you may use something like that:
>>>>>
>>>>> iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m conntrack
>>>>> --cstate NEW,RELATED -j MARK --set-mark 1
>>>>>
>>>>> The main idea consists in marking packets on the physical input
>>>>> interface and shaping them on ifb0( where they arrive already marked
>>>>> ).
>>>>> iptables' packet marks exist only in memory of one computer, TOS/DSCP
>>>>> may be used for transmitting a map of them to another one.
>>>>> BTW, use --restore-mark on the output interface of your shaper too.
>>>>>
>>>>> 2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
>>>>>> Thanks a lot for your help, how can I evaluate --restore-mark before
>>>>>> I
>>>>>> classify and shape response traffic from the origin server?
>>>>>>
>>>>>> I think you mean something like:
>>>>>>
>>>>>>      # Copy ctmart to nfmark (e.g. 1, 2)
>>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK
>>>>>> --restore-mark
>>>>>>
>>>>>>      # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class
>>>>>> 2:2
>>>>>>      tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw
>>>>>> flowid
>>>>>> 2:1 action mirred egress redirect dev ifb0
>>>>>>      tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw
>>>>>> flowid
>>>>>> 2:3 action mirred egress redirect dev ifb0
>>>>>>      tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0
>>>>>> 0
>>>>>> flowid 2:2 action mirred egress redirect dev ifb0
>>>>>>
>>>>>> Just how can I get --restore-mark to evaluate before tc filter?
>>>>>>
>>>>>> Another way I can imagine is with the CLASSIFY target:
>>>>>>
>>>>>>      # Send unmarked traffic to class 2:2
>>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1
>>>>>> -j
>>>>>> CLASSIFY 2:1
>>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2
>>>>>> -j
>>>>>> CLASSIFY 2:3
>>>>>>      iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>>>>>>
>>>>>> But I have the same challenge, how can I evaluate the CLASSIFY target
>>>>>> before I shape traffic?
>>>>>>
>>>>>> Or is there another way to classify and shape response traffic from
>>>>>> the
>>>>>> origin server based on the TOS/DSCP field of the request?
>>>>>>
>>>>>> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>>>>>>> You use iptables mark + restore mark based on connection tracking.
>>>>>>> you can mark the TOS on the outgoing postrouting table.
>>>>>>> you can take a look at the iptabes man pages:
>>>>>>> http://ipset.netfilter.org/iptables.man.html
>>>>>>> which has --restore-mark  exaple.
>>>>>>>
>>>>>>> Eliezer
>>>>>>>
>>>>>>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>>>>>>> I can imagine a couple ways of classifying traffic from our proxy
>>>>>>>> server
>>>>>>>> based on the TOS/DSCP field, and also how to set the connection
>>>>>>>> mark
>>>>>>>> based on this field. But how do I classify and shape response
>>>>>>>> traffic
>>>>>>>> from the origin server based on the connection mark?
>>>>>> --
>>>>>> To unsubscribe from this list: send the line "unsubscribe netfilter"
>>>>>> in
>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>>
>


-- 
Best regards
Anatoly Muliarski

Re: Mark traffic on one machine, match on another machine?

[Jack Bates]

Thanks for this advice Anatoly, I corrected the script to redirect all 
incoming traffic to ifb0, and now statistics on the ifb0 classes and 
qdiscs are incrementing, but they're still not incrementing in the way I 
expect. Outgoing traffic is still getting classified as expected: If I 
make a request with TOS 28, I watch the statistics of eth0.2 class 2:3 
increment, but the response causes ifb0 class 2:2 to increment, not 2:3.

I think that the TOS field is correctly matched and the mark is 
correctly saved and restored, but only outgoing traffic is getting 
classified based on the mark? Do you have any more suggestions how to 
investigate why incoming traffic isn't getting classified? or is there 
another way to classify response traffic based on the TOS/DSCP field of 
the request?

Here is the corrected script:

    1: iptables -N NEW_CONN -t mangle
    2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark 1
    3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark 2
    4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK --set-mark 3
    5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
    6:
    7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID 
-j DROP
    8: iptables -A PREROUTING -t mangle -m conntrack --ctstate 
ESTABLISHED -j CONNMARK --restore-mark
    9: iptables -A PREROUTING -t mangle -m conntrack --ctstate 
NEW,RELATED -j NEW_CONN
   10:
   11: ifconfig ifb0 up
   12:
   13: insmod sch_ingress
   14: tc qdisc add dev eth0.2 ingress
   15:
   16: insmod cls_u32
   17: insmod act_mirred
   18: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1 
u32 match u32 0 0 action mirred egress redirect dev ifb0
   19:
   20: # Limit up and downstream to 1mbit, to "own" the queue
   21: insmod sch_tbf
   22: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k 
latency 70ms
   23: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k 
latency 70ms
   24:
   25: # High, low, and "other" priority
   26: insmod sch_prio
   27: tc qdisc add dev eth0.2 parent 1: handle 2 prio
   28: tc qdisc add dev ifb0 parent 1: handle 2 prio
   29:
   30: insmod cls_fw
   31:
   32: # Classify request traffic from proxy server based on mark
   33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw 
flowid :1
   34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw 
flowid :3
   35: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 3 fw 
flowid :2
   36:
   37: # Classify response traffic from origin server based on restored mark
   38: tc filter add dev ifb0 parent 2: protocol ip pref 1 handle 1 fw 
flowid :1
   39: tc filter add dev ifb0 parent 2: protocol ip pref 1 handle 2 fw 
flowid :3
   40: tc filter add dev ifb0 parent 2: protocol ip pref 1 handle 3 fw 
flowid :2

On 10/12/12 12:11 PM, Anatoly Muliarski wrote:
> IMHO, the problem lies in next - incoming packets on eth0.2 have no
> mark and therefore they cannot be forwarded into ifb0 on the current
> classification scheme. You should redirect all incoming packets( or
> use non-mark redirection criteria ) into ifb0 and then classify and
> shape them on ifb0 device.
>
> To more clarify the way of a packet look at
> http://en.wikipedia.org/wiki/File:Netfilter-packet-flow.svg
>
> 2012/12/10, Jack Bates <uo4zau@nottheoilrig.com>:
>> Thank you for this clarification, and your assumptions are correct: I'm
>> trying to prioritize incoming and outgoing traffic on eth0.2, ifb0 is
>> for shaping the incoming traffic.
>>
>> But I'm still failing to filter incoming traffic based on the restored
>> mark: statistics on the ifb0 qdiscs and classes remain zero. Filtering
>> outgoing traffic is working as expected, statistics on the eth0.2
>> classes are incrementing as expected, which I think means that the
>> PREROUTING --restore-mark target is working?
>>
>> Do you have any other suggestions how to investigate why incoming
>> traffic isn't getting filtered? or is there another way to classify
>> response traffic from the origin server based on the TOS/DSCP field of
>> the request from the proxy server? Are there any more details that would
>> help explain the problem?
>>
>> Here is the clarified script:
>>
>>      1: iptables -N NEW_CONN -t mangle
>>      2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark 1
>>      3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark 2
>>      4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK --set-mark 3
>>      5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
>>      6:
>>      7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID
>> -j DROP
>>      8: iptables -A PREROUTING -t mangle -m conntrack --ctstate
>> ESTABLISHED -j CONNMARK --restore-mark
>>      9: iptables -A PREROUTING -t mangle -m conntrack --ctstate
>> NEW,RELATED -j NEW_CONN
>>     10:
>>     11: ifconfig ifb0 up
>>     12:
>>     13: insmod sch_ingress
>>     14: tc qdisc add dev eth0.2 ingress
>>     15:
>>     16: # Classify response traffic from origin server based on restored
>> mark
>>     17: insmod cls_fw
>>     18: insmod act_mirred
>>     19: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1
>> fw flowid 2:1 action mirred egress redirect dev ifb0
>>     20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 2
>> fw flowid 2:3 action mirred egress redirect dev ifb0
>>     21: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 3
>> fw flowid 2:2 action mirred egress redirect dev ifb0
>>     22:
>>     23: # Limit up and downstream to 1mbit, to "own" the queue
>>     24: insmod sch_tbf
>>     25: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
>> latency 70ms
>>     26: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
>> latency 70ms
>>     27:
>>     28: # High, low, and "other" priority
>>     29: insmod sch_prio
>>     30: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>>     31: tc qdisc add dev ifb0 parent 1: handle 2 prio
>>     32:
>>     33: # Classify request traffic from proxy server based on mark
>>     34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw
>> flowid :1
>>     35: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw
>> flowid :3
>>     36: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 3 fw
>> flowid :2
>>
>> On 05/12/12 08:18 PM, Anatoly Muliarski wrote:
>>> IMHO, there is a bit of mess in your scheme.
>>> Let's clarify it :)
>>>
>>> If my assumptions are right then:
>>> 1. eth0.2 is the external shaper interface.
>>> 2. ifb0 is using for shaping incoming traffic on this interface.
>>>
>>> To set marks for the traffic you should use something like these:
>>>
>>> iptables -t manlge -N NEW_CONN
>>> iptables -t mangle -A NEW_CONN -m tos --tos 12 -j MARK --set-mark 1
>>> iptables -t mangle -A NEW_CONN -m tos --tos 28 -j MARK --set-mark 2
>>> # to mark all the unmatched traffic and treat it separately in your
>>> shaper
>>> iptables -t mangle -A NEW_CONN -m mark --mark 0  -j MARK --set-mark 3
>>> iptables -t mangle -A NEW_CONN  -j CONNMARK --save-mark
>>>
>>> iptables -t mangle -A PREROUTING -m conntrack --cstate INVALID -j DROP
>>> iptables -t mangle -A PREROUTING -m conntrack --cstate ESTABLISHED -j
>>> CONNMARK --restore-mark
>>> iptables -t mangle -A PREROUTING -m conntrack --cstate NEW,RELATED -g
>>> NEW_CONN
>>>
>>>
>>> 2012/12/5, Jack Bates <uo4zau@nottheoilrig.com>:
>>>> Thanks for this help, I think the marks are getting saved/restored
>>>> successfully, and I can successfully classify request traffic from the
>>>> proxy server based on the mark (statistics on the eth0.2 classes are
>>>> incrementing) but response traffic from the origin server isn't getting
>>>> classified yet (statistics on the ifb0 classes remain zero).
>>>>
>>>> Here is my complete work in progress script. How can I investigate why
>>>> response traffic isn't getting classified? Are there any more details
>>>> that would help explain why not?
>>>>
>>>> How can I classify traffic based on the restored mark? or is there
>>>> another way to classify response traffic from the origin server based on
>>>> the TOS/DSCP field of the request?
>>>>
>>>>       1: # Save/restore connection mark based on TOS field
>>>>       2: iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
>>>>       3: iptables -A PREROUTING -t mangle -m tos --tos 12 -j MARK
>>>> --set-mark
>>>> 1
>>>>       4: iptables -A PREROUTING -t mangle -m tos --tos 28 -j MARK
>>>> --set-mark
>>>> 2
>>>>       5: iptables -A PREROUTING -t mangle -j CONNMARK --save-mark
>>>>       6:
>>>>       7: ifconfig ifb0 up
>>>>       8:
>>>>       9: insmod sch_ingress
>>>>      10: tc qdisc add dev eth0.2 ingress
>>>>      11:
>>>>      12: # Classify response traffic from origin server based on restored
>>>> mark
>>>>      13: insmod cls_fw
>>>>      14: insmod act_mirred
>>>>      15: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle
>>>> 1
>>>> fw flowid 2:1 action mirred egress redirect dev ifb0
>>>>      16: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle
>>>> 2
>>>> fw flowid 2:3 action mirred egress redirect dev ifb0
>>>>      17:
>>>>      18: # Send unmarked traffic to class 2:2
>>>>      19: insmod cls_u32
>>>>      20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 u32
>>>> match u32 0 0 flowid 2:2 action mirred egress redirect dev ifb0
>>>>      21:
>>>>      22: # Limit up and downstream to 1mbit, to "own" the queue
>>>>      23: insmod sch_tbf
>>>>      24: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
>>>> latency 70ms
>>>>      25: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
>>>> latency 70ms
>>>>      26:
>>>>      27: # High, low, and "other" priority
>>>>      28: insmod sch_prio
>>>>      29: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>>>>      30: tc qdisc add dev ifb0 parent 1: handle 2 prio
>>>>      31:
>>>>      32: # Classify request traffic from proxy server based on mark
>>>>      33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1
>>>> fw
>>>> flowid :1
>>>>      34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2
>>>> fw
>>>> flowid :3
>>>>
>>>> On 05/12/12 01:12 AM, Eliezer Croitoru wrote:
>>>>> Thanks Anatoly,
>>>>>
>>>>> My idea was that based on a mark he will jump the packets to another
>>>>> table which HE will mark TOS/DSCP.
>>>>>
>>>>> Eliezer
>>>>>
>>>>> On 12/5/2012 4:39 AM, Anatoly Muliarski wrote:
>>>>>> Hi Jack,
>>>>>>
>>>>>> --restore-mark should be used for existing connections and to mark new
>>>>>> ones you may use something like that:
>>>>>>
>>>>>> iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m conntrack
>>>>>> --cstate NEW,RELATED -j MARK --set-mark 1
>>>>>>
>>>>>> The main idea consists in marking packets on the physical input
>>>>>> interface and shaping them on ifb0( where they arrive already marked
>>>>>> ).
>>>>>> iptables' packet marks exist only in memory of one computer, TOS/DSCP
>>>>>> may be used for transmitting a map of them to another one.
>>>>>> BTW, use --restore-mark on the output interface of your shaper too.
>>>>>>
>>>>>> 2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
>>>>>>> Thanks a lot for your help, how can I evaluate --restore-mark before
>>>>>>> I
>>>>>>> classify and shape response traffic from the origin server?
>>>>>>>
>>>>>>> I think you mean something like:
>>>>>>>
>>>>>>>       # Copy ctmart to nfmark (e.g. 1, 2)
>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK
>>>>>>> --restore-mark
>>>>>>>
>>>>>>>       # Classify by nfmark (e.g. 1, 2), send unmarked traffic to class
>>>>>>> 2:2
>>>>>>>       tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw
>>>>>>> flowid
>>>>>>> 2:1 action mirred egress redirect dev ifb0
>>>>>>>       tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw
>>>>>>> flowid
>>>>>>> 2:3 action mirred egress redirect dev ifb0
>>>>>>>       tc filter add dev eth0.2 parent ffff: protocol ip u32 match u32 0
>>>>>>> 0
>>>>>>> flowid 2:2 action mirred egress redirect dev ifb0
>>>>>>>
>>>>>>> Just how can I get --restore-mark to evaluate before tc filter?
>>>>>>>
>>>>>>> Another way I can imagine is with the CLASSIFY target:
>>>>>>>
>>>>>>>       # Send unmarked traffic to class 2:2
>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 1
>>>>>>> -j
>>>>>>> CLASSIFY 2:1
>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark 2
>>>>>>> -j
>>>>>>> CLASSIFY 2:3
>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>>>>>>>
>>>>>>> But I have the same challenge, how can I evaluate the CLASSIFY target
>>>>>>> before I shape traffic?
>>>>>>>
>>>>>>> Or is there another way to classify and shape response traffic from
>>>>>>> the
>>>>>>> origin server based on the TOS/DSCP field of the request?
>>>>>>>
>>>>>>> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>>>>>>>> You use iptables mark + restore mark based on connection tracking.
>>>>>>>> you can mark the TOS on the outgoing postrouting table.
>>>>>>>> you can take a look at the iptabes man pages:
>>>>>>>> http://ipset.netfilter.org/iptables.man.html
>>>>>>>> which has --restore-mark  exaple.
>>>>>>>>
>>>>>>>> Eliezer
>>>>>>>>
>>>>>>>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>>>>>>>> I can imagine a couple ways of classifying traffic from our proxy
>>>>>>>>> server
>>>>>>>>> based on the TOS/DSCP field, and also how to set the connection
>>>>>>>>> mark
>>>>>>>>> based on this field. But how do I classify and shape response
>>>>>>>>> traffic
>>>>>>>>> from the origin server based on the connection mark?
>>>>>>> --
>>>>>>> To unsubscribe from this list: send the line "unsubscribe netfilter"
>>>>>>> in
>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>>
>>
>
>

Re: Mark traffic on one machine, match on another machine?

[Anatoly Muliarski]

Quickly, I have 2 ideas:
1. Check if you load conntrack module( or conntrack support is
built-in in your kernel )
2. For a short period of time or for specific conditions try to use:
iptables -t raw -A PREROUTING ( ... here is the conditions ...) -j TRACE
Then use can analyze your syslog and see the way of packet flow.

2012/12/12, Jack Bates <uo4zau@nottheoilrig.com>:
> Thanks for this advice Anatoly, I corrected the script to redirect all
> incoming traffic to ifb0, and now statistics on the ifb0 classes and
> qdiscs are incrementing, but they're still not incrementing in the way I
> expect. Outgoing traffic is still getting classified as expected: If I
> make a request with TOS 28, I watch the statistics of eth0.2 class 2:3
> increment, but the response causes ifb0 class 2:2 to increment, not 2:3.
>
> I think that the TOS field is correctly matched and the mark is
> correctly saved and restored, but only outgoing traffic is getting
> classified based on the mark? Do you have any more suggestions how to
> investigate why incoming traffic isn't getting classified? or is there
> another way to classify response traffic based on the TOS/DSCP field of
> the request?
>
> Here is the corrected script:
>
>     1: iptables -N NEW_CONN -t mangle
>     2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark 1
>     3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark 2
>     4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK --set-mark 3
>     5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
>     6:
>     7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID
> -j DROP
>     8: iptables -A PREROUTING -t mangle -m conntrack --ctstate
> ESTABLISHED -j CONNMARK --restore-mark
>     9: iptables -A PREROUTING -t mangle -m conntrack --ctstate
> NEW,RELATED -j NEW_CONN
>    10:
>    11: ifconfig ifb0 up
>    12:
>    13: insmod sch_ingress
>    14: tc qdisc add dev eth0.2 ingress
>    15:
>    16: insmod cls_u32
>    17: insmod act_mirred
>    18: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle 1
> u32 match u32 0 0 action mirred egress redirect dev ifb0
>    19:
>    20: # Limit up and downstream to 1mbit, to "own" the queue
>    21: insmod sch_tbf
>    22: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
> latency 70ms
>    23: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
> latency 70ms
>    24:
>    25: # High, low, and "other" priority
>    26: insmod sch_prio
>    27: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>    28: tc qdisc add dev ifb0 parent 1: handle 2 prio
>    29:
>    30: insmod cls_fw
>    31:
>    32: # Classify request traffic from proxy server based on mark
>    33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1 fw
> flowid :1
>    34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2 fw
> flowid :3
>    35: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 3 fw
> flowid :2
>    36:
>    37: # Classify response traffic from origin server based on restored
> mark
>    38: tc filter add dev ifb0 parent 2: protocol ip pref 1 handle 1 fw
> flowid :1
>    39: tc filter add dev ifb0 parent 2: protocol ip pref 1 handle 2 fw
> flowid :3
>    40: tc filter add dev ifb0 parent 2: protocol ip pref 1 handle 3 fw
> flowid :2
>
> On 10/12/12 12:11 PM, Anatoly Muliarski wrote:
>> IMHO, the problem lies in next - incoming packets on eth0.2 have no
>> mark and therefore they cannot be forwarded into ifb0 on the current
>> classification scheme. You should redirect all incoming packets( or
>> use non-mark redirection criteria ) into ifb0 and then classify and
>> shape them on ifb0 device.
>>
>> To more clarify the way of a packet look at
>> http://en.wikipedia.org/wiki/File:Netfilter-packet-flow.svg
>>
>> 2012/12/10, Jack Bates <uo4zau@nottheoilrig.com>:
>>> Thank you for this clarification, and your assumptions are correct: I'm
>>> trying to prioritize incoming and outgoing traffic on eth0.2, ifb0 is
>>> for shaping the incoming traffic.
>>>
>>> But I'm still failing to filter incoming traffic based on the restored
>>> mark: statistics on the ifb0 qdiscs and classes remain zero. Filtering
>>> outgoing traffic is working as expected, statistics on the eth0.2
>>> classes are incrementing as expected, which I think means that the
>>> PREROUTING --restore-mark target is working?
>>>
>>> Do you have any other suggestions how to investigate why incoming
>>> traffic isn't getting filtered? or is there another way to classify
>>> response traffic from the origin server based on the TOS/DSCP field of
>>> the request from the proxy server? Are there any more details that would
>>> help explain the problem?
>>>
>>> Here is the clarified script:
>>>
>>>      1: iptables -N NEW_CONN -t mangle
>>>      2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark
>>> 1
>>>      3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark
>>> 2
>>>      4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK
>>> --set-mark 3
>>>      5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
>>>      6:
>>>      7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID
>>> -j DROP
>>>      8: iptables -A PREROUTING -t mangle -m conntrack --ctstate
>>> ESTABLISHED -j CONNMARK --restore-mark
>>>      9: iptables -A PREROUTING -t mangle -m conntrack --ctstate
>>> NEW,RELATED -j NEW_CONN
>>>     10:
>>>     11: ifconfig ifb0 up
>>>     12:
>>>     13: insmod sch_ingress
>>>     14: tc qdisc add dev eth0.2 ingress
>>>     15:
>>>     16: # Classify response traffic from origin server based on restored
>>> mark
>>>     17: insmod cls_fw
>>>     18: insmod act_mirred
>>>     19: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle
>>> 1
>>> fw flowid 2:1 action mirred egress redirect dev ifb0
>>>     20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle
>>> 2
>>> fw flowid 2:3 action mirred egress redirect dev ifb0
>>>     21: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 handle
>>> 3
>>> fw flowid 2:2 action mirred egress redirect dev ifb0
>>>     22:
>>>     23: # Limit up and downstream to 1mbit, to "own" the queue
>>>     24: insmod sch_tbf
>>>     25: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
>>> latency 70ms
>>>     26: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
>>> latency 70ms
>>>     27:
>>>     28: # High, low, and "other" priority
>>>     29: insmod sch_prio
>>>     30: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>>>     31: tc qdisc add dev ifb0 parent 1: handle 2 prio
>>>     32:
>>>     33: # Classify request traffic from proxy server based on mark
>>>     34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 1
>>> fw
>>> flowid :1
>>>     35: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 2
>>> fw
>>> flowid :3
>>>     36: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle 3
>>> fw
>>> flowid :2
>>>
>>> On 05/12/12 08:18 PM, Anatoly Muliarski wrote:
>>>> IMHO, there is a bit of mess in your scheme.
>>>> Let's clarify it :)
>>>>
>>>> If my assumptions are right then:
>>>> 1. eth0.2 is the external shaper interface.
>>>> 2. ifb0 is using for shaping incoming traffic on this interface.
>>>>
>>>> To set marks for the traffic you should use something like these:
>>>>
>>>> iptables -t manlge -N NEW_CONN
>>>> iptables -t mangle -A NEW_CONN -m tos --tos 12 -j MARK --set-mark 1
>>>> iptables -t mangle -A NEW_CONN -m tos --tos 28 -j MARK --set-mark 2
>>>> # to mark all the unmatched traffic and treat it separately in your
>>>> shaper
>>>> iptables -t mangle -A NEW_CONN -m mark --mark 0  -j MARK --set-mark 3
>>>> iptables -t mangle -A NEW_CONN  -j CONNMARK --save-mark
>>>>
>>>> iptables -t mangle -A PREROUTING -m conntrack --cstate INVALID -j DROP
>>>> iptables -t mangle -A PREROUTING -m conntrack --cstate ESTABLISHED -j
>>>> CONNMARK --restore-mark
>>>> iptables -t mangle -A PREROUTING -m conntrack --cstate NEW,RELATED -g
>>>> NEW_CONN
>>>>
>>>>
>>>> 2012/12/5, Jack Bates <uo4zau@nottheoilrig.com>:
>>>>> Thanks for this help, I think the marks are getting saved/restored
>>>>> successfully, and I can successfully classify request traffic from the
>>>>> proxy server based on the mark (statistics on the eth0.2 classes are
>>>>> incrementing) but response traffic from the origin server isn't
>>>>> getting
>>>>> classified yet (statistics on the ifb0 classes remain zero).
>>>>>
>>>>> Here is my complete work in progress script. How can I investigate why
>>>>> response traffic isn't getting classified? Are there any more details
>>>>> that would help explain why not?
>>>>>
>>>>> How can I classify traffic based on the restored mark? or is there
>>>>> another way to classify response traffic from the origin server based
>>>>> on
>>>>> the TOS/DSCP field of the request?
>>>>>
>>>>>       1: # Save/restore connection mark based on TOS field
>>>>>       2: iptables -A PREROUTING -t mangle -j CONNMARK --restore-mark
>>>>>       3: iptables -A PREROUTING -t mangle -m tos --tos 12 -j MARK
>>>>> --set-mark
>>>>> 1
>>>>>       4: iptables -A PREROUTING -t mangle -m tos --tos 28 -j MARK
>>>>> --set-mark
>>>>> 2
>>>>>       5: iptables -A PREROUTING -t mangle -j CONNMARK --save-mark
>>>>>       6:
>>>>>       7: ifconfig ifb0 up
>>>>>       8:
>>>>>       9: insmod sch_ingress
>>>>>      10: tc qdisc add dev eth0.2 ingress
>>>>>      11:
>>>>>      12: # Classify response traffic from origin server based on
>>>>> restored
>>>>> mark
>>>>>      13: insmod cls_fw
>>>>>      14: insmod act_mirred
>>>>>      15: tc filter add dev eth0.2 parent ffff: protocol ip pref 1
>>>>> handle
>>>>> 1
>>>>> fw flowid 2:1 action mirred egress redirect dev ifb0
>>>>>      16: tc filter add dev eth0.2 parent ffff: protocol ip pref 1
>>>>> handle
>>>>> 2
>>>>> fw flowid 2:3 action mirred egress redirect dev ifb0
>>>>>      17:
>>>>>      18: # Send unmarked traffic to class 2:2
>>>>>      19: insmod cls_u32
>>>>>      20: tc filter add dev eth0.2 parent ffff: protocol ip pref 1 u32
>>>>> match u32 0 0 flowid 2:2 action mirred egress redirect dev ifb0
>>>>>      21:
>>>>>      22: # Limit up and downstream to 1mbit, to "own" the queue
>>>>>      23: insmod sch_tbf
>>>>>      24: tc qdisc add dev eth0.2 root handle 1 tbf rate 1mbit burst 5k
>>>>> latency 70ms
>>>>>      25: tc qdisc add dev ifb0 root handle 1 tbf rate 1mbit burst 5k
>>>>> latency 70ms
>>>>>      26:
>>>>>      27: # High, low, and "other" priority
>>>>>      28: insmod sch_prio
>>>>>      29: tc qdisc add dev eth0.2 parent 1: handle 2 prio
>>>>>      30: tc qdisc add dev ifb0 parent 1: handle 2 prio
>>>>>      31:
>>>>>      32: # Classify request traffic from proxy server based on mark
>>>>>      33: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle
>>>>> 1
>>>>> fw
>>>>> flowid :1
>>>>>      34: tc filter add dev eth0.2 parent 2: protocol ip pref 1 handle
>>>>> 2
>>>>> fw
>>>>> flowid :3
>>>>>
>>>>> On 05/12/12 01:12 AM, Eliezer Croitoru wrote:
>>>>>> Thanks Anatoly,
>>>>>>
>>>>>> My idea was that based on a mark he will jump the packets to another
>>>>>> table which HE will mark TOS/DSCP.
>>>>>>
>>>>>> Eliezer
>>>>>>
>>>>>> On 12/5/2012 4:39 AM, Anatoly Muliarski wrote:
>>>>>>> Hi Jack,
>>>>>>>
>>>>>>> --restore-mark should be used for existing connections and to mark
>>>>>>> new
>>>>>>> ones you may use something like that:
>>>>>>>
>>>>>>> iptables -t mangle -A PREROUTING -i eth0.2 -m tos --tos 1 -m
>>>>>>> conntrack
>>>>>>> --cstate NEW,RELATED -j MARK --set-mark 1
>>>>>>>
>>>>>>> The main idea consists in marking packets on the physical input
>>>>>>> interface and shaping them on ifb0( where they arrive already marked
>>>>>>> ).
>>>>>>> iptables' packet marks exist only in memory of one computer,
>>>>>>> TOS/DSCP
>>>>>>> may be used for transmitting a map of them to another one.
>>>>>>> BTW, use --restore-mark on the output interface of your shaper too.
>>>>>>>
>>>>>>> 2012/12/3, Jack Bates <uo4zau@nottheoilrig.com>:
>>>>>>>> Thanks a lot for your help, how can I evaluate --restore-mark
>>>>>>>> before
>>>>>>>> I
>>>>>>>> classify and shape response traffic from the origin server?
>>>>>>>>
>>>>>>>> I think you mean something like:
>>>>>>>>
>>>>>>>>       # Copy ctmart to nfmark (e.g. 1, 2)
>>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -j CONNMARK
>>>>>>>> --restore-mark
>>>>>>>>
>>>>>>>>       # Classify by nfmark (e.g. 1, 2), send unmarked traffic to
>>>>>>>> class
>>>>>>>> 2:2
>>>>>>>>       tc filter add dev eth0.2 parent ffff: protocol ip handle 1 fw
>>>>>>>> flowid
>>>>>>>> 2:1 action mirred egress redirect dev ifb0
>>>>>>>>       tc filter add dev eth0.2 parent ffff: protocol ip handle 2 fw
>>>>>>>> flowid
>>>>>>>> 2:3 action mirred egress redirect dev ifb0
>>>>>>>>       tc filter add dev eth0.2 parent ffff: protocol ip u32 match
>>>>>>>> u32 0
>>>>>>>> 0
>>>>>>>> flowid 2:2 action mirred egress redirect dev ifb0
>>>>>>>>
>>>>>>>> Just how can I get --restore-mark to evaluate before tc filter?
>>>>>>>>
>>>>>>>> Another way I can imagine is with the CLASSIFY target:
>>>>>>>>
>>>>>>>>       # Send unmarked traffic to class 2:2
>>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark
>>>>>>>> 1
>>>>>>>> -j
>>>>>>>> CLASSIFY 2:1
>>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -m connmark --mark
>>>>>>>> 2
>>>>>>>> -j
>>>>>>>> CLASSIFY 2:3
>>>>>>>>       iptables -A PREROUTING -t mangle -i eth0.2 -j CLASSIFY 2:2
>>>>>>>>
>>>>>>>> But I have the same challenge, how can I evaluate the CLASSIFY
>>>>>>>> target
>>>>>>>> before I shape traffic?
>>>>>>>>
>>>>>>>> Or is there another way to classify and shape response traffic from
>>>>>>>> the
>>>>>>>> origin server based on the TOS/DSCP field of the request?
>>>>>>>>
>>>>>>>> On 03/12/12 03:52 AM, Eliezer Croitoru wrote:
>>>>>>>>> You use iptables mark + restore mark based on connection tracking.
>>>>>>>>> you can mark the TOS on the outgoing postrouting table.
>>>>>>>>> you can take a look at the iptabes man pages:
>>>>>>>>> http://ipset.netfilter.org/iptables.man.html
>>>>>>>>> which has --restore-mark  exaple.
>>>>>>>>>
>>>>>>>>> Eliezer
>>>>>>>>>
>>>>>>>>> On 12/3/2012 10:43 AM, Jack Bates wrote:
>>>>>>>>>> I can imagine a couple ways of classifying traffic from our proxy
>>>>>>>>>> server
>>>>>>>>>> based on the TOS/DSCP field, and also how to set the connection
>>>>>>>>>> mark
>>>>>>>>>> based on this field. But how do I classify and shape response
>>>>>>>>>> traffic
>>>>>>>>>> from the origin server based on the connection mark?
>>>>>>>> --
>>>>>>>> To unsubscribe from this list: send the line "unsubscribe
>>>>>>>> netfilter"
>>>>>>>> in
>>>>>>>> the body of a message to majordomo@vger.kernel.org
>>>>>>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>


-- 
Best regards
Anatoly Muliarski

Re: Mark traffic on one machine, match on another machine?

[Andrew Collins]

> Here is the corrected script:
>
>
>    1: iptables -N NEW_CONN -t mangle
>    2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark 1
>    3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark 2
>    4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK --set-mark 3
>    5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
>    6:
>    7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID -j
> DROP
>    8: iptables -A PREROUTING -t mangle -m conntrack --ctstate ESTABLISHED -j
> CONNMARK --restore-mark
>    9: iptables -A PREROUTING -t mangle -m conntrack --ctstate NEW,RELATED -j
> NEW_CONN

Ingress qdisc traffic is handled before conntrack has seen the
traffic, so the mark you're restoring hasn't actually been restored at
the time when your classifier rules are executing.  It's a general
problem that makes ingress shaping based upon connection information
difficult to achieve.

The only way I've found to solve this is to classify directly in TC
without the use of conntrack/marks, or use IMQ
(http://www.linuximq.net/).  Perhaps there's a clever way to hook into
connmark with act_ipt, but I haven't found it.

Re: Mark traffic on one machine, match on another machine?

[Anatoly Muliarski]

2012/12/13, Andrew Collins <bsderandrew@gmail.com>:
>> Here is the corrected script:
>>
>>
>>    1: iptables -N NEW_CONN -t mangle
>>    2: iptables -A NEW_CONN -t mangle -m tos --tos 12 -j MARK --set-mark 1
>>    3: iptables -A NEW_CONN -t mangle -m tos --tos 28 -j MARK --set-mark 2
>>    4: iptables -A NEW_CONN -t mangle -m mark --mark 0 -j MARK --set-mark
>> 3
>>    5: iptables -A NEW_CONN -t mangle -j CONNMARK --save-mark
>>    6:
>>    7: iptables -A PREROUTING -t mangle -m conntrack --ctstate INVALID -j
>> DROP
>>    8: iptables -A PREROUTING -t mangle -m conntrack --ctstate ESTABLISHED
>> -j
>> CONNMARK --restore-mark
>>    9: iptables -A PREROUTING -t mangle -m conntrack --ctstate NEW,RELATED
>> -j
>> NEW_CONN
>
> Ingress qdisc traffic is handled before conntrack has seen the
> traffic, so the mark you're restoring hasn't actually been restored at
> the time when your classifier rules are executing.  It's a general
> problem that makes ingress shaping based upon connection information
> difficult to achieve.
>
> The only way I've found to solve this is to classify directly in TC
> without the use of conntrack/marks, or use IMQ
> (http://www.linuximq.net/).  Perhaps there's a clever way to hook into
> connmark with act_ipt, but I haven't found it.
>

The traffic will be ingress on eth0.2, but it will be treated as
egress on ifb0 device.
Using ifb is a common scheme to overcome the above-mentioned limitation.



-- 
Best regards
Anatoly Muliarski

Re: Mark traffic on one machine, match on another machine?

[Andrew Collins]

> The traffic will be ingress on eth0.2, but it will be treated as
> egress on ifb0 device.
> Using ifb is a common scheme to overcome the above-mentioned limitation.

From the point of view of queueing/TC this is true, however it will
not pass through the neftilter hooks while going through the IFB, so
iptables/conntrack will still have no chance to see it until after
it's already egressed the qdisc attached to the IFB.

Re: Mark traffic on one machine, match on another machine?

[Anatoly Muliarski]

2012/12/14, Andrew Collins <bsderandrew@gmail.com>:
>> The traffic will be ingress on eth0.2, but it will be treated as
>> egress on ifb0 device.
>> Using ifb is a common scheme to overcome the above-mentioned limitation.
>
> From the point of view of queueing/TC this is true, however it will
> not pass through the neftilter hooks while going through the IFB, so
> iptables/conntrack will still have no chance to see it until after
> it's already egressed the qdisc attached to the IFB.
>

To my shame, you are right.
The IFB device has no netfilter hooks, so we can not use iptables to
mark traffic on it.
The external incoming traffic should be shaped on the internal
outgoing interface.


-- 
Best regards
Anatoly Muliarski

Re: Mark traffic on one machine, match on another machine?

[Jan Engelhardt]

On Wednesday 2012-11-28 06:09, Jack Bates wrote:

> Is there a way to mark traffic on one machine (proxy server) and match the mark
> on another machine (router)?

The Flowlabel field in IPv6 would lend itself to that..