[LARTC] Strategy for about 200 part-time users

[LARTC] Strategy for about 200 part-time users

[Jan Wilson]

I've been lurking for a while, trying to figure out this traffic
control stuff.  We have 3 ADSL modems upstream, with 2, 2, and 1
megabit total bandwidth, 1/4 of it up, 3/4 down.

We also have about 200 workstations downstream, connected through
wifi.  We are presently using tc to create about 200 buckets with HTB
and SFQ.  It seems like this is too many buckets, since usually only
about 80 of the 200 are active, meaning the bucket size for each
workstation is small.

We have trouble with P2P uploads, but the P2P solutions I have seen,
if I understand them, get rather unwieldy if you have to keep a couple
hundred clients separate rather than grouping them by protocol, etc.

We absolutely need to be able to keep RRD stats on bandwidth usage per
client, but we also need to switch from IP to MAC-based access.  It
would really help if I could get a few comments on my feeble attempts
to develop a strategy here:

Does it sound feasible to keep a database of MAC addresses and client
numbers and flush/regenerate tc buckets about every other minute?

First, iptables would check authorized MAC addresses and mark packets
with a client number.  This client number would be the basis for
throttling uploads.

I would run a Perl script that would fetch the ARP table to see what
MAC addresses are on, and pair them with a client number and an IP
number.  In some cases there would be multiple IPs with the same MAC,
or multiple IPs and MACs that are the same client, depending on the
router/bridge in use.

The Perl script would run maybe once every other minute and
flush/recreate HTB/SFQ buckets for all current active clients, plus
one "misc" bucket, that would get freshly logged on clients, and the
occasional client who swapped their wifi card without telling us.

Does this strategy seem ... reasonable?  Other solutions?  Thanks lots.

-- 
Jan Wilson, SysAdmin     _/*];          jan@corozal.com
Corozal Junior College   |  |:'  corozal.com corozal.bz
Corozal Town, Belize     |  /'  chetumal.com & linux.bz
Reg. Linux user #151611  |_/   Network, PHP, Perl, HTML
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Ed Wildgoose]

>Does this strategy seem ... reasonable?  Other solutions?  Thanks lots.
>  
>

What is your goal?  Why isn't 2 buckets, one for P2P and the everything 
else bucket (with an SFQ on each) enough for what you want (ie what's 
your target?)  What's wrong with the current setup (apart from wanting 
to switch to MAC access)?

I think I saw that there was something called ESFQ which does something 
similar to SFQ but on a per IP basis, rather than a per port/IP basis.  
Perhaps that would help?  The HOWTO has an example of setting up a hash 
type ruleset, perhaps that could be modified to work with MAC addresses...?

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Jan Wilson]

* Ed Wildgoose <lists@wildgooses.com> [040518 08:57]:
> What is your goal?  Why isn't 2 buckets, one for P2P and the everything 
> else bucket (with an SFQ on each) enough for what you want (ie what's 
> your target?)  What's wrong with the current setup (apart from wanting 
> to switch to MAC access)?

Well, as I see it (which could easily be wrong), the problem is with
setting up over 200 buckets, even though only 60 to 90, maybe, are
active at any one time.  With the limited bandwidth we have available
(it is very expensive here), it makes rather tiny buckets.

If we further divide so there is a bucket for P2P and a regular bucket
for each client, now you have over 600 buckets, right?  One class per
user and two leaves? Unfortunately, we HAVE to break this down by
client, and be able to track bandwidth by client.  In fact, by client,
rather than by machine, which is why I thought of marking by a client
number, on upload, at least.  Then, on download, IP numbers could
translate BACK into client marks by the same Perl script and lookups.

What I was really asking is whether redistributing the buckets every 2
minutes is a problem.  I know the SFQ perturbs every 5 or 10 seconds,
so 2 minutes seemed conservative, comparatively.

But I know I might be looking at this all wrong, so I'd appreciate any
hint I could get.

> I think I saw that there was something called ESFQ which does something 
> similar to SFQ but on a per IP basis, rather than a per port/IP basis.  
> Perhaps that would help?  The HOWTO has an example of setting up a hash 
> type ruleset, perhaps that could be modified to work with MAC addresses...?

Thanks, I'll take a look at that also.  While the traffic control
documentation is pretty good, it could use more commented examples.

-- 
Jan Wilson, SysAdmin     _/*];          jan@corozal.com
Corozal Junior College   |  |:'  corozal.com corozal.bz
Corozal Town, Belize     |  /'  chetumal.com & linux.bz
Reg. Linux user #151611  |_/   Network, PHP, Perl, HTML
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Andreas Klauer]

Am Tuesday 18 May 2004 17:42 schrieb Jan Wilson:
> Unfortunately, we HAVE to break this down by client, and be able to track
> bandwidth by client.  In fact, by client, rather than by machine

If I get this right, you want to know how much traffic each client caused,
because they have to pay for it. This should be done independent of your
actual traffic shaping strategy. You shouldn't need tc to count traffic?

> What I was really asking is whether redistributing the buckets every 2
> minutes is a problem.  I know the SFQ perturbs every 5 or 10 seconds,
> so 2 minutes seemed conservative, comparatively.

You could add/remove classes on the fly as required. You probably should 
avoid removing all classes everytime a client becomes (in)active. But if 
you want to enhance quality and P2P traffic really is a problem, I don't 
see another way than to move this traffic away from the user classes.

In my own script, I put P2P into user class, because it's intended for 
small home network. Imagine you have 5 users, even if 4 have P2P running 
to max the line out, the 5th user still gets 1/5th of bandwidth 
guaranteed, which is probably enough. With 80 user, he'd only get 1/80th, 
which probably isn't enough to do anything useful. Of course, it also 
depends on the total bandwidth of your line and your client's 
requirements.

I've never worked with big networks, but I imagine that P2P traffic there 
is pretty much unwanted. So why do you want to allow P2P traffic eating 
99% of your line? Sure, if you limit P2P to say 25% of your line, paying 
users may complain about bad P2P performance - but that's still better 
than all users complaining about bad overall performance in general.

Andreas
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Jan Wilson]

* Andreas Klauer <Andreas.Klauer@metamorpher.de> [040518 10:32]:
> If I get this right, you want to know how much traffic each client caused,
> because they have to pay for it. This should be done independent of your
> actual traffic shaping strategy. You shouldn't need tc to count traffic?

Actually, we charge most clients the same, but we would like to
restrict the bandwidth hogs somewhat, and in a throttling situation,
be able to give some priority to clients who "behave".

Do you mean that we should NOT be using the tc to count traffic?  What
instead?  We are using a Python script that reads values using
tc -s -d class show <device>

Is there a better way?

> You could add/remove classes on the fly as required. You probably should 
> avoid removing all classes everytime a client becomes (in)active. But if 
> you want to enhance quality and P2P traffic really is a problem, I don't 
> see another way than to move this traffic away from the user classes.

Maybe a compromise ... I could have the script set up x number of
buckets and add/delete up to or down to a certain number of classes,
then flush all and recalculate and redistribute the bandwidth only
after the number of classes has exceeded or fallen below a certain
number.

> In my own script, I put P2P into user class, because it's intended for 
> small home network. Imagine you have 5 users, even if 4 have P2P running 
> to max the line out, the 5th user still gets 1/5th of bandwidth 
> guaranteed, which is probably enough. With 80 user, he'd only get 1/80th, 
> which probably isn't enough to do anything useful. Of course, it also 
> depends on the total bandwidth of your line and your client's 
> requirements.

We don't want to just drop all P2P packets, so I guess we will have to
throttle it, since our P2P users will suck up all the upload that is
available, 24 hours a day if allowed.  We would like to place the
heavy uploaders into a tighter upload bucket, without regard to
whether the upload is P2P or something else (don't know what).

What I've seen on this list is that if we have to double or triple our
HTB buckets in order to use ipp2p, it might not be worth it for a
large network with limited bandwidth.  If this is not accurate, I hope
someone will correct me.

> I've never worked with big networks, but I imagine that P2P traffic there 
> is pretty much unwanted. So why do you want to allow P2P traffic eating 
> 99% of your line? Sure, if you limit P2P to say 25% of your line, paying 
> users may complain about bad P2P performance - but that's still better 
> than all users complaining about bad overall performance in general.

What we would prefer is for the P2P user to complain because he/she is
not downloading very fast, and then we can explain it is because they
leave P2P running ... that may lead to some social engineering.

Our bandwidth is SO expensive, we want to avoid adding additional
bandwidth as long as possible.  It is also why we are using ADSL
rather than T1 or something.

Thanks for your comments.

-- 
Jan Wilson, SysAdmin     _/*];          jan@corozal.com
Corozal Junior College   |  |:'  corozal.com corozal.bz
Corozal Town, Belize     |  /'  chetumal.com & linux.bz
Reg. Linux user #151611  |_/   Network, PHP, Perl, HTML
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Ed Wildgoose]

Jan Wilson wrote:

>* Ed Wildgoose <lists@wildgooses.com> [040518 08:57]:
>  
>
>>What is your goal?  Why isn't 2 buckets, one for P2P and the everything 
>>else bucket (with an SFQ on each) enough for what you want (ie what's 
>>your target?)  What's wrong with the current setup (apart from wanting 
>>to switch to MAC access)?
>>    
>>
>
>Well, as I see it (which could easily be wrong), the problem is with
>setting up over 200 buckets, even though only 60 to 90, maybe, are
>active at any one time.  With the limited bandwidth we have available
>(it is very expensive here), it makes rather tiny buckets.
>  
>

Yeah, but the buckets have a guaranteed bandwidth, and also get to share 
the remaining bandwidth between them.  So if you have 600 buckets, and 
only two are active, then those two buckets should get half the 
bandwidth each (if you set it up that way).  Perhaps that's really 
obvious, but just wanted to be sure

Also, I think you can do your accounting with firewall rules as well as 
tc rules.  It may be more flexible to seperate accounting from 
throttling - perhaps it will give you more options to set this up.

For example, you could setup the rules with the class of traffic at the 
top and the users underneath, eg top level is "P2P", "bulk", 
"interactive", and under each is a user class (or SFQ, etc).  Now you 
could limit the P2P to be only 1/3 of total bandwidth, which is then 
fought over by the users (each will get a fraction depending on how many 
others are using it, not how many classes you have).  Perhaps for the 
other classes even SFQ will be enough? 

The howto mentions some examples of using hashing which may allow you a 
concise way to setup these rules by IP for example (even though you 
won't be accounting that way).

You then need to look to iptables to see if you can do some standard 
rules to account for your traffic.  Perhaps someone here could suggest a 
neat way?

Good luck.  Would be interested to hear your final solution

Ed W
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Jan Wilson]

* Ed Wildgoose <lists@wildgooses.com> [040518 14:20]:
> Jan Wilson wrote:
> >Well, as I see it (which could easily be wrong), the problem is with
> >setting up over 200 buckets, even though only 60 to 90, maybe, are
> >active at any one time.  With the limited bandwidth we have available
> >(it is very expensive here), it makes rather tiny buckets.
> 
> Yeah, but the buckets have a guaranteed bandwidth, and also get to share 
> the remaining bandwidth between them.  So if you have 600 buckets, and 
> only two are active, then those two buckets should get half the 
> bandwidth each (if you set it up that way).  Perhaps that's really 
> obvious, but just wanted to be sure

I never mind anyone stating the "obvious" ... one person's obvious is
another's "eureka"  ;-)

But I find it awkward, for our purposes, the way rate and ceil seem to
work.  If we theoretically have 200 clients, and all 200 are on, then
we can set rate for 1/200 of the total ... in our case only about 10
kilobits per user.  They CAN borrow up to ceil, yes.  But since we are
REALLY busy, we can't really do much to give one client more than
another.

However, if we KNOW only 50 are on right now, then we can give each of
those a rate of 1/50, guaranteed.  We can set ceil a little
differently.  And if we have bandwidth hogs, we can adjust things so
it is fairer for all.

Suppose we have invited 200 guests for dessert.  So we make 200 tarts
to be sure everyone gets one.  But only 50 guests show up.  So they
eat their own tart, and then fight over the other 150 tarts.

What if you could look at the total number of guests before they enter
the dining room, make a quick assessment, and put two or three tarts
on each plate instead of one.  They still might squabble a bit over
the remaining tarts, but not as much of a free-for-all as before.

That is what I was thinking to do with my Perl script.

> Also, I think you can do your accounting with firewall rules as well as 
> tc rules.  It may be more flexible to seperate accounting from 
> throttling - perhaps it will give you more options to set this up.

This is a really interesting idea ... if we can keep our accounting
separate then it WOULD open up more possibilities.  Maybe someone can
point me in the direction of IP traffic accounting that we could use
separate from the tc.

> For example, you could setup the rules with the class of traffic at the 
> top and the users underneath, eg top level is "P2P", "bulk", 
> "interactive", and under each is a user class (or SFQ, etc).  Now you 
> could limit the P2P to be only 1/3 of total bandwidth, which is then 
> fought over by the users (each will get a fraction depending on how many 
> others are using it, not how many classes you have).  Perhaps for the 
> other classes even SFQ will be enough? 

I think I can see that ... if we don't have to put each customer into
one (or two or three) different buckets.  If we could even put groups
of customers into buckets together, that might be good.  But again, it
seems like it would be handy to be able to dynamically change bucket
sizes and whose packets go through them.

> The howto mentions some examples of using hashing which may allow you a 
> concise way to setup these rules by IP for example (even though you 
> won't be accounting that way).

I looked at that again, and I do see that it might be useful,
especially if we can handle our bandwidth accounting separately.  If
anyone has a working example I could look at, I might actually have a
chance of understanding it  ;-)  I have no trouble with the hex math,
being an old assembly language programmer, but it isn't quite sinking
in.

> You then need to look to iptables to see if you can do some standard 
> rules to account for your traffic.  Perhaps someone here could suggest a 
> neat way?

That would be REALLY helpful.
 
> Good luck.  Would be interested to hear your final solution

I promise there will be a link to a well-documented Perl script when I
get something working.  :-)  Even if it is a static arrangement, I
want to set it up with a script that can be adjusted easily when we
have new customers, MAC address changes, additional ADSL modems, etc.

Thanks lots for your suggestions.

-- 
Jan Wilson, SysAdmin     _/*];          jan@corozal.com
Corozal Junior College   |  |:'  corozal.com corozal.bz
Corozal Town, Belize     |  /'  chetumal.com & linux.bz
Reg. Linux user #151611  |_/   Network, PHP, Perl, HTML
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Ed Wildgoose]

>But I find it awkward, for our purposes, the way rate and ceil seem to
>work.  If we theoretically have 200 clients, and all 200 are on, then
>we can set rate for 1/200 of the total ... in our case only about 10
>kilobits per user.  They CAN borrow up to ceil, yes.  But since we are
>REALLY busy, we can't really do much to give one client more than
>another.
>  
>

Hey, I just read the docs, I don't have any experience of using this.  
BUT, there appears to be a priority option (it's not called priority, 
but I don't have the docs in front of me) on the HTB qdisc which seems 
to let you bias the squabble towards certain users and away from 
others.  I vaguely recall that it might be that the excess gets divided 
as a ratio of the priorities?

>However, if we KNOW only 50 are on right now, then we can give each of
>those a rate of 1/50, guaranteed.  We can set ceil a little
>differently.  And if we have bandwidth hogs, we can adjust things so
>it is fairer for all.
>
>Suppose we have invited 200 guests for dessert.  So we make 200 tarts
>to be sure everyone gets one.  But only 50 guests show up.  So they
>eat their own tart, and then fight over the other 150 tarts.
>
>What if you could look at the total number of guests before they enter
>the dining room, make a quick assessment, and put two or three tarts
>on each plate instead of one.  They still might squabble a bit over
>the remaining tarts, but not as much of a free-for-all as before.
>  
>

Hmm, perhaps I'm being stupid, but those two situations seem to be 
identical to me?  I can't see the difference between 1 "tart", and a 
1/50 share of the remaining 150 tarts, or starting with 1/50 of 200 
tarts...?

Also, you appear to be more interested in priortising based on the 
*type* of traffic.  So that appears to be more logically at the top of 
your tree.  Its only like this that you can limit the P2P people I 
think.  Otherwise all you can say is that each P2P user can use their 
full allocation of bandwidth to download junk.  However, if you put P2P 
at the top and limit it to 20% of total, then each user will be able to 
use *up to* their full quota, but if there are lots of P2P users, then 
each may not even be able to reach that level, and so will find they are 
throttled back below even their personal download quota.  This appears 
to be what you are after?


>This is a really interesting idea ... if we can keep our accounting
>separate then it WOULD open up more possibilities.  Maybe someone can
>point me in the direction of IP traffic accounting that we could use
>separate from the tc.
>  
>

I thought that there were accounting logs on all iptables chains.  There 
is also an accounting option in iptables I seem to remember (check man 
pages).  Perhaps there are other options involving a user space queue 
which doesn't actually queue, but counts stuff...  I'm sure there are 
other options as well

>>For example, you could setup the rules with the class of traffic at the 
>>top and the users underneath, eg top level is "P2P", "bulk", 
>>"interactive", and under each is a user class (or SFQ, etc).  Now you 
>>could limit the P2P to be only 1/3 of total bandwidth, which is then 
>>fought over by the users (each will get a fraction depending on how many 
>>others are using it, not how many classes you have).  Perhaps for the 
>>other classes even SFQ will be enough? 
>>    
>>
>
>I think I can see that ... if we don't have to put each customer into
>one (or two or three) different buckets.  If we could even put groups
>of customers into buckets together, that might be good.  But again, it
>seems like it would be handy to be able to dynamically change bucket
>sizes and whose packets go through them.
>  
>

I think this method will make that easier?  Remember that each customer 
will turn up multiple times now, ie for each type of traffic.  You can 
more easily tune the buckets determining the amount of each sort of 
traffic, eg turning up P2P in the evening, and down in the day simply by 
adjusting the single top bucket (not each individual one).

Good luck

Ed W
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Jason Boxman]

On Tuesday 18 May 2004 17:41, Jan Wilson wrote:
<snip>
> I promise there will be a link to a well-documented Perl script when I
> get something working.  :-)  Even if it is a static arrangement, I
> want to set it up with a script that can be adjusted easily when we
> have new customers, MAC address changes, additional ADSL modems, etc.

Then indeed it does sound like you have two different but related problem sets 
here.  1) You want to do bandwidth accounting for customers.  2) You want 
traffic shaping to provide the smoothest service possible.

The former must have some search hits on Google I'd imagine.  Bandwidth 
accounting is a popular thing.

> Thanks lots for your suggestions.

I have been following your posts with interest.  I look forward to hearing 
about the solutions you devise.

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Andreas Klauer]

Am Tuesday 18 May 2004 19:43 schrieb Jan Wilson:
> Maybe a compromise ... I could have the script set up x number of
> buckets and add/delete up to or down to a certain number of classes,
> then flush all and recalculate and redistribute the bandwidth only
> after the number of classes has exceeded or fallen below a certain
> number.

Well - HTB will work too even if the children rates added together don't
equal the parent rate (Wondershaper is living proof ;-P), so probably 
recreating all classes is not necessary. It's more important that there 
are as few classes as possible (no class for inactive user).

So you could try just adding a class when a user gets active and removing a 
class when a user gets inactive.  I think there was someone on this list 
who had a script that worked like that with ~1000 users in total.

People notice in my LAN (which has only 5 users) when I re-run my fairnat 
script (the machine it runs on is quite old, so it takes several second 
until everything is re-created, and interactive connections tend to get 
sloppy for a while after that), so I think it would be really bad if you 
did the same every 2 or somewhat minutes with so many users.

> We would like to place the heavy uploaders into a tighter upload bucket,
> without regard to whether the upload is P2P or something else (don't know
> what). 

Sure, that sounds like a good idea.
However, what exactly do you intend to do to 'tighten' things?

> What I've seen on this list is that if we have to double or triple our
> HTB buckets in order to use ipp2p, it might not be worth it for a
> large network with limited bandwidth.

Shaping on a user basis will only affect the P2P users. If you make an 
IPP2P class as child of the user class with low priority, the effect will 
be that the P2P user can still use interactive connections (because his 
own P2P traffic has lower priority). But for other users it won't matter, 
they don't know wether the traffic caused by users is P2P or something 
else. The only thing you could do on a user basis would be setting a hard 
ceil for P2P. That way, P2P traffic of a user could never cause the user 
class to borrow. But I don't recommend this setup - prioritizing 
interactive over P2P traffic is something the user should do at home.

> Our bandwidth is SO expensive, we want to avoid adding additional
> bandwidth as long as possible.  It is also why we are using ADSL
> rather than T1 or something.

Adding bandwidth wouldn't change much anyway, as long as P2P and similar 
takes all the bandwidth there is. So this way or another you need a good 
solution to clean this traffic mess up.

HTH
Andreas
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Strategy for about 200 part-time users

[Andy Furniss]

Jan Wilson wrote:
> I've been lurking for a while, trying to figure out this traffic
> control stuff.  We have 3 ADSL modems upstream, with 2, 2, and 1
> megabit total bandwidth, 1/4 of it up, 3/4 down.

Once you have your strategy/policy sorted the detail here could be 
important. What are the bitrates of each connection, are you 
loadbalancing or choosing your own routes, what interface do you (plan 
to) shape on?

Andy.

<snip>

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/