IPMARK

IPMARK

[Jason Cosby]

All,

Sorry to post here, but the LARTC list has gone belly up. I have a LAN of 35 with IPs from 192.168.1.6 thru 192.168.1.40 sharing a satellite link. I need to limit total bandwidth to 512Kbps and divide it evenly between all 35 clients. My plan is to create 35 individual queues and allow 100% borrowing so that the queues don't have to constantly be deleted/recreated upon user enter/exit. All traffic is run through NAT at the gateway computer. 

I see using IPMARK as the way to go, but am not clear how to put it together. The documentation doesn't quite clear it up for me. Can anyone help me get together a simple down and dirty script to do as I described? Once I get the network under control a bit I will continue implementing proper QOS. If I don't get a handle on this soon I will be tarred, feathered, and thrown in the desert to rot (almost), so any help is GREATLY appreciated. 

Thanks,
Jason

Re: IPMARK

[ArcosCom Linux User]

Why you want to use IPMARK?

If you only want traffic priorization, perhaps you are looking for
"wondershapper":
   http://lartc.org/wondershaper/
   http://freshmeat.net/projects/wshaper/

Or take a view into:

http://lartc.org/howto/lartc.cookbook.ultimate-tc.html

Another project you can take a view (and is more recent), is this:

http://www.adsl-optimizer.dk/

Regards

El Jue, 18 de Septiembre de 2008, 10:30, Jason Cosby escribió:
> All,
>
> Sorry to post here, but the LARTC list has gone belly up. I have a LAN of
> 35 with IPs from 192.168.1.6 thru 192.168.1.40 sharing a satellite link. I
> need to limit total bandwidth to 512Kbps and divide it evenly between all
> 35 clients. My plan is to create 35 individual queues and allow 100%
> borrowing so that the queues don't have to constantly be deleted/recreated
> upon user enter/exit. All traffic is run through NAT at the gateway
> computer.
>
> I see using IPMARK as the way to go, but am not clear how to put it
> together. The documentation doesn't quite clear it up for me. Can anyone
> help me get together a simple down and dirty script to do as I described?
> Once I get the network under control a bit I will continue implementing
> proper QOS. If I don't get a handle on this soon I will be tarred,
> feathered, and thrown in the desert to rot (almost), so any help is
> GREATLY appreciated.
>
> Thanks,
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: IPMARK

[julien vehent]

2008/9/18 Jason Cosby <sky_jason@yahoo.com>:
> All,
>
> Sorry to post here, but the LARTC list has gone belly up. I have a LAN of 35 with IPs from 192.168.1.6 thru 192.168.1.40 sharing a satellite link. I need to limit total bandwidth to 512Kbps and divide it evenly between all 35 clients. My plan is to create 35 individual queues and allow 100% borrowing so that the queues don't have to constantly be deleted/recreated upon user enter/exit. All traffic is run through NAT at the gateway computer.
>

I think if you just create one queue that is limited to 512kbps and
redirect the 35 clients to this queue, the outgoing traffic will be
divided evenly between them.

Try this (assuming your network card is eth0):

tc qdisc add dev eth0 root handle 1: htb default 10
tc class add dev eth0 parent 1:1 classid 1:10 htb rate 512kbps
tc filter add dev eth0 parent 1:0 protocol ip prio 1 handle 1 fw flowid 1:10
iptables -t mangle -A PREROUTING -s 192.168.1.0/24 -j MARK --set-mark 1



> I see using IPMARK as the way to go, but am not clear how to put it together. The documentation doesn't quite clear it up for me. Can anyone help me get together a simple down and dirty script to do as I described? Once I get the network under control a bit I will continue implementing proper QOS. If I don't get a handle on this soon I will be tarred, feathered, and thrown in the desert to rot (almost), so any help is GREATLY appreciated.
>
> Thanks,
> Jason
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: IPMARK

[Michal Soltys]

Jason Cosby wrote:
> All,
> 
> I see using IPMARK as the way to go, but am not clear how to put it 
> together. The documentation doesn't quite clear it up for me. Can 
> anyone help me get together a simple down and dirty script to do as I
>  described? Once I get the network under control a bit I will
> continue implementing proper QOS. If I don't get a handle on this
> soon I will be tarred, feathered, and thrown in the desert to rot
> (almost), so any help is GREATLY appreciated.
> 

You can as well use CLASSIFY target, instead of MARK+tc filter.

Example (pretty crude and simplified - there're many factors to consider
after all) of what you have in mind:


#adjust queue lengths in "pfifo" to your needs
#adjust interface name
eth=eth1

tc qdisc add dev $eth root handle 1:0 hfsc default 101
tc class add dev $eth parent 1:0 classid 1:1 hfsc ls m2 512kbps \
         ul m2 512kbps

#default queue
tc class add dev $eth parent 1:1 classid 1:101 hfsc rt m2 60kbps \
         ls m2 200kbps
tc qdisc add dev $eth handle 101:0 parent 1:101 pfifo limit 10

#client queues
tc class add dev $eth parent 1:1 classid 1:102 hfsc rt m2 400kbps \
         ls m2 400kbps
tc qdisc add dev $eth handle 102:0 parent 1:102 sfq limit 20 \
         perturb 10 quantum 1

iptables -t mangle -A FORWARD -o $eth -m iprange --src-range \
         192.168.1.6-192.168.1.40 -j CLASSIFY --set-class 1:102


The above will guarantee 60kbps for all the other traffic, 400kbps for
the clients, and divide any unused bandwidth in 1:2 ratio (in hfsc -
realtime guerantees use actual values, but linkshare criterion based on
ratios of values - 200:400 here). SFQ will take care of distributing the
bandwidth across all the clients. Check out recently added flow 
classifier to extend SFQ functionality similary to what was possible 
with ESFQ ( http://marc.info/?l=linux-netdev&m=120180241422360&w=2 ).

Alternatively - you could create 35 hfsc leaf classes in a loop, but
getting sizes of the leaf queues can become tricky (there may be
something I'm not aware of though), and SFQ seems like a much better 
thing (especially with mentioned above functionality extended by 'flow').

E.g. something along the lines of:

for i in `seq -w 6 1 40` ; do
         iptables -t mangle -A MARKCHAIN -s 192.168.1.1${i} \
                 -j CLASSIFY --set-class 1:1${i}
         tc class add dev $eth parent 1:1 classid 1:1${i} \
                 hfsc sc m2 10000
         tc qdisc add dev $eth handle 1${i}:0 parent 1:1${i} \
                 pfifo limit 3
done

If you gonna read about traffic shaping, be sure to check out:

http://ace-host.stuart.id.au/russell/files/tc/doc/

It's the very good source of info for u32 filter, among other things 
(old lartc faq is very innacurate here).

Re: IPMARK

[julien vehent]

So CLASSIFY can do it too...
Are you aware of any performance difference between MARK/tc filter and
CLASSIFY ?


2008/9/19 Michal Soltys <nozo@ziu.info>:
> Jason Cosby wrote:
>>
>> All,
>>
>> I see using IPMARK as the way to go, but am not clear how to put it
>> together. The documentation doesn't quite clear it up for me. Can anyone
>> help me get together a simple down and dirty script to do as I
>>  described? Once I get the network under control a bit I will
>> continue implementing proper QOS. If I don't get a handle on this
>> soon I will be tarred, feathered, and thrown in the desert to rot
>> (almost), so any help is GREATLY appreciated.
>>
>
> You can as well use CLASSIFY target, instead of MARK+tc filter.
>
> Example (pretty crude and simplified - there're many factors to consider
> after all) of what you have in mind:
>
>
> #adjust queue lengths in "pfifo" to your needs
> #adjust interface name
> eth=eth1
>
> tc qdisc add dev $eth root handle 1:0 hfsc default 101
> tc class add dev $eth parent 1:0 classid 1:1 hfsc ls m2 512kbps \
>        ul m2 512kbps
>
> #default queue
> tc class add dev $eth parent 1:1 classid 1:101 hfsc rt m2 60kbps \
>        ls m2 200kbps
> tc qdisc add dev $eth handle 101:0 parent 1:101 pfifo limit 10
>
> #client queues
> tc class add dev $eth parent 1:1 classid 1:102 hfsc rt m2 400kbps \
>        ls m2 400kbps
> tc qdisc add dev $eth handle 102:0 parent 1:102 sfq limit 20 \
>        perturb 10 quantum 1
>
> iptables -t mangle -A FORWARD -o $eth -m iprange --src-range \
>        192.168.1.6-192.168.1.40 -j CLASSIFY --set-class 1:102
>
>
> The above will guarantee 60kbps for all the other traffic, 400kbps for
> the clients, and divide any unused bandwidth in 1:2 ratio (in hfsc -
> realtime guerantees use actual values, but linkshare criterion based on
> ratios of values - 200:400 here). SFQ will take care of distributing the
> bandwidth across all the clients. Check out recently added flow classifier
> to extend SFQ functionality similary to what was possible with ESFQ (
> http://marc.info/?l=linux-netdev&m=120180241422360&w=2 ).
>
> Alternatively - you could create 35 hfsc leaf classes in a loop, but
> getting sizes of the leaf queues can become tricky (there may be
> something I'm not aware of though), and SFQ seems like a much better thing
> (especially with mentioned above functionality extended by 'flow').
>
> E.g. something along the lines of:
>
> for i in `seq -w 6 1 40` ; do
>        iptables -t mangle -A MARKCHAIN -s 192.168.1.1${i} \
>                -j CLASSIFY --set-class 1:1${i}
>        tc class add dev $eth parent 1:1 classid 1:1${i} \
>                hfsc sc m2 10000
>        tc qdisc add dev $eth handle 1${i}:0 parent 1:1${i} \
>                pfifo limit 3
> done
>
> If you gonna read about traffic shaping, be sure to check out:
>
> http://ace-host.stuart.id.au/russell/files/tc/doc/
>
> It's the very good source of info for u32 filter, among other things (old
> lartc faq is very innacurate here).
>
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: IPMARK

[Michal Soltys]

julien vehent wrote:
> So CLASSIFY can do it too...
> Are you aware of any performance difference between MARK/tc filter and
> CLASSIFY ?
> 

Not really. Both MARK and CLASSIFY mainly differ by the field they set
(respectively: skb->mark and skb->priority, after quick peek over the 
code). So - not much of a difference between those two, except that MARK 
allows for some extra functionality.

IPMARK

[Jason Cosby]

Thanks for the responses. I've studied Wondershaper, the LARTC, and ADSL-optimizer. There are lots of great ideas in there for TOS bandwidth division (also important), but, AFAIK, FairNAT is the only one focused on per user bandwidth division. 

Currently, I've got the user mark derived from the last part of the IP address * 10. It would save a lot of work and scripting using IPMARK, and I was getting errors out of MARK that I never figured out (might be the lack of using hex). Right now, I also have to specify the IPs in the script. I was hoping that using IPMARK would eliminate this as well--i.e. whatever IP comes along is checked against allowed MACs and auto assigned a user mark without me having to worry about it.

I'm also not clear on and'ing the user mark with the TOS mark after I get a basic per user bandwidth division going on and am iplementing TOS marking. 

Thanks,
Jason 

IPMARK

[Jason Cosby]

Thank you so much for taking the time to help me out. If I could pick this apart a bit more and understand it, I would be on my way. 

eth=eth1--this refers to LAN dev or NET dev? (I ran the first line via ssh below on LAN dev and it locked up the machine. This is for egress then? I neglected to mention that ingress is what I seek to control initially, which will require IMQ AFAIK. When finished, I'll have done the math and controlled egress to the point where ingress is very close to where it needs to be without policing.) 

tc class add dev $eth parent 1:0 classid 1:1 hfsc ls m2 512kbps \
ul m2 512kbps

ls=link sharing, ul=upper limit, clear on those. m2 is synonimous with sc?


tc class add dev $eth parent 1:1 classid 1:101 hfsc rt m2 60kbps \
ls m2 200kbps

rt=realtime, clear on that. Not clear on 200kbps spec. Related to upper limit or can borrow up to 200?

tc qdisc add dev $eth handle 101:0 parent 1:101 pfifo limit 10

pfifo because we don't need anything more advanced here, we don't know what kind of traffic we're catching, don't know destination for IP based queue, catching fragments, or some other reason?


tc class add dev $eth parent 1:1 classid 1:102 hfsc rt m2 400kbps \
ls m2 400kbps

I'm not tracking on how we can have 400kbps of realtime and linksharing simultaneously. They're not mutually exclusive? Not sure what the 1:2 ratio (200:400) translates to, but I know that understanding this is vital. 

tc qdisc add dev $eth handle 102:0 parent 1:102 sfq limit 20 perturb 10 quantum 1

How did we arrive at limit of 20? quantum 1 is to ensure maximum granularity vs. a higher number?

iptables -t mangle -A FORWARD -o $eth -m iprange --src-range 192.168.1.6-192.168.1.40 -j CLASSIFY --set-class 1:102

This is gold and what I was searching for (before hfsc got my interest). Makes perfect sense. 

Thanks so much. Hopefully others will benefit from the light being shed on hfsc here.

Jason

Re: IPMARK

[Michal Soltys]

Jason Cosby wrote:
> Thank you so much for taking the time to help me out. If I could pick
> this apart a bit more and understand it, I would be on my way.
> 
> eth=eth1--this refers to LAN dev or NET dev? (I ran the first line
> via ssh below on LAN dev and it locked up the machine. This is for
> egress then? I neglected to mention that ingress is what I seek to
> control initially, which will require IMQ AFAIK. When finished, I'll
> have done the math and controlled egress to the point where ingress
> is very close to where it needs to be without policing.)
> 

Yes, I meant egress. For ingress you need IMQ, or these days - IFB ( 
http://www.linuxfoundation.org/en/Net:IFB ). Controlling on ingress and 
how much point is in it (as bandwidth is already wasted), is a 
controversial subject :). Besides, as it's your router - you can shape 
on egress on the other side, unless you want to alter traffic coming to 
router itself.

> tc class add dev $eth parent 1:0 classid 1:1 hfsc ls m2 512kbps \ ul
> m2 512kbps
> 
> ls=link sharing, ul=upper limit, clear on those. m2 is synonimous
> with sc?
> 

m2 is the slope of the second part of the curve (ls or rt one). m1 
(unused here) is the first one. It allows for non-linear curve 
specification, when initial delay is important (e.g. in voip). Sc (short 
for service curve) can be used instead of specifying rt and ls (when the 
same value is used).

> tc class add dev $eth parent 1:1 classid 1:101 hfsc rt m2 60kbps \ ls
>  m2 200kbps
> 
> rt=realtime, clear on that. Not clear on 200kbps spec. Related to 
> upper limit or can borrow up to 200?
> 
> I'm not tracking on how we can have 400kbps of realtime and 
> linksharing simultaneously. They're not mutually exclusive? Not sure
>  what the 1:2 ratio (200:400) translates to, but I know that 
> understanding this is vital.
> 

Realtime (rt) and linkshare (ls) curves are independent. First always rt 
is considered, and class hierarchy is irrelevant here - only leafs are 
checked. Only and only after that - linksharing and class hierarchy is 
considered. Concept of borrowing doesn't exist in hfsc.

I can imagine it's a bit messy explanation without the basics about 
hfsc. It's probably best to read about the whole thing, check out:

http://www.cs.cmu.edu/~istoica/hfsc-tr.ps.gz

For more gentle introductions:

http://www.sonycsl.co.jp/~kjc/software/TIPS.txt (section related to 
hfsc, but note this is BSD implementation specific - afaik, there's no 
80% bw limit for realtime in linux, and I'm not sure if convex curves 
are implemented the same [simplified] way. Either way, it's pretty nice 
intro).

http://linux-ip.net/articles/hfsc.en/
http://marc.info/?t=107799591400001&r=1&w=2

> 
> pfifo because we don't need anything more advanced here, we don't
> know what kind of traffic we're catching, don't know destination for
> IP based queue, catching fragments, or some other reason?
> 

pfifo (or bfifo), because if you don't specify it explicitly, queue 
length will be taken from interface's txqueuelen. You can of course 
alter it with ip link set dev eth0 txqueuelen <nbr>, but that's the 
default for all leaf qdiscs not specified explicitly (they get 
pfifo_fast with interface's txqueuelen). Well - whatever qdisc you use, 
is up to you - for the sake of this example pfifo was a good choice ;)


> tc qdisc add dev $eth handle 102:0 parent 1:102 sfq limit 20 perturb
> 10 quantum 1
> 
> How did we arrive at limit of 20? quantum 1 is to ensure maximum
> granularity vs. a higher number?
> 

check: man tc-sfq , tc qdisc add sfq help, and 
http://ace-host.stuart.id.au/russell/files/tc/doc/sch_sfq.txt

Remember about mentioned 'tc filter flow' in this case.

As for the queue length - well it depends on many things - bandwidth, HZ 
of your kernel (to be honest, I don't know how tickless (NO_HZ) option 
influences this - anyone could shed some light on it ?), how much delay 
is acceptable, how the typical traffic looks like in your case, etc.

> iptables -t mangle -A FORWARD -o $eth -m iprange --src-range
> 192.168.1.6-192.168.1.40 -j CLASSIFY --set-class 1:102
> 
> This is gold and what I was searching for (before hfsc got my
> interest). Makes perfect sense.
> 

Don't forget about IPSET and CONNMARK which can give a great deal of 
extra flexibility.