[LARTC] Too stupid to figure out shaping

[LARTC] Too stupid to figure out shaping

[John Gorkos]

First I'll confess my sins, then I'll beg for help.

I own a small wireless ISP, and I sell service at three levels, 256kb/s, 
384kb/s, and 512kb/s.  For about 18 months, I thought I had this bandwidth 
limiting figured out.  I had three HTB classes off the root, one each with 
the limits above.  Since each customer has a single IP address, I used 
iptables to mark packets destined for each subscriber with level 1,2 or 3.  
It looked like it worked great:  Customer A got 256, B got 384, and C got his 
512.  In hindsight, it was wishful thinking, since ALL 256kb/s customers got 
dumped into the same 256 class and had to duke it out for that 256kb/s, 
instead of each customer getting their own 256kb/s slice.  Eventually, as I 
added more customers, people started complaining that they weren't getting 
what they're paying for (rightly).

So, now I'm running my ISP with no bandwidth shaping and I'm struggling to get 
my brain wrapped around how make sure everyone gets what they pay for, but 
not more.

From the reading I've done, it looks like I need a separate class for each 
subscriber.  Inside that class, I'd like to have a standard set of queues to 
prioritize each customer's slice of bandwidth by port (typical three band 
stuff:  interactive, web,bulk).
  So assuming I've got three customers:
10.0.0.10 gets256kb/s
10.0.0.11 gets 386 kb/s
10.0.0.12 gets 256kb/s
 I think I'd have a tree like this:

                        CLASS 10 (256kb/s) (inner classes prioritize)
ROOT -----        CLASS 11(384kb/s)                 "
                        CLASS 12 (256kb/s)                "
plus three filters to direct iptables-marked traffic to the appropriate queue 
and three iptables entries to mark the traffic appropriately.

The problem is, I'm simply not smart enough to actually IMPLEMENT this.  I 
tried setting up the HTB classes, and when I added the 5th one (class 15 in 
the little diagram above), my interactivity to the router went to near 1000ms 
RTT (I was ssh'ed into it).  I started getting calls from my customers 
immediately saying something was 'wrong with the internet'.  I'm not sure 
what happened, since I hadn't installed any filters or anything.

Anyway, I'd dearly appreciate some help on this.  Surely this is a nut that 
someone has already cracked, but for the life of me all I can find on the 
internet are how-to shape your own outbound traffic to your ISP so your P2P 
traffic (the BANE of ISPs) doesn't interfere with your Doom3 deathmatch.

Thanks in advance,
John Gorkos
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Too stupid to figure out shaping

[David Boreham]

I also run a small wireless ISP, and we use HTB for shaping.
You need a separate class for each customer.
Alternatively I think it might be ok to use SFQ with
multiple customers in the same class, but for us the
class per customer works fine.

Fundamentally you need some mechanism that ensures
that each customer's packets are queued in their own queue.
Otherwise they will interfere with each other's QoS.


_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Too stupid to figure out shaping

[Brian Carrig]

John,

Personally I think you're doing too much. All the ISPs here offer a specified amount of 
bandwidth to different classes of customer. However, they also list a contention ratio. 
Thus they might say customer class A gets 512Kbit/s with a contention rate of 8:1 
and customer class B gets 256Kbit/s with a contention ratio of 12:1.

This is very simple to set up. Assume we have eight class A customers and twelve 
class B customers. Similar to what you did before, set up two HTB classes off the 
root and use ip tables to mark customer packets according to their assigned IP 
address. Make sure that the HTB classes are using SFQ. Now all eight class A 
customers will share the 512Kbit/s class, with traffic distributed evenly among all 
active flows.

If this type of solution is not satisfactory for your customers then you will need to set 
up a separate class for each individual customer and mark their packets 
appropriately. This may be more hassle than its worth, depending on the number of 
customers you have.

Regards
Brian Carrig

On 27 Apr 2005 at 10:35, John Gorkos wrote:

> First I'll confess my sins, then I'll beg for help.
> 
> I own a small wireless ISP, and I sell service at three levels,
> 256kb/s, 384kb/s, and 512kb/s.  For about 18 months, I thought I had
> this bandwidth limiting figured out.  I had three HTB classes off the
> root, one each with the limits above.  Since each customer has a
> single IP address, I used iptables to mark packets destined for each
> subscriber with level 1,2 or 3.  It looked like it worked great: 
> Customer A got 256, B got 384, and C got his 512.  In hindsight, it
> was wishful thinking, since ALL 256kb/s customers got dumped into the
> same 256 class and had to duke it out for that 256kb/s, instead of
> each customer getting their own 256kb/s slice.  Eventually, as I added
> more customers, people started complaining that they weren't getting
> what they're paying for (rightly).
> 
> So, now I'm running my ISP with no bandwidth shaping and I'm
> struggling to get my brain wrapped around how make sure everyone gets
> what they pay for, but not more.
> 
> From the reading I've done, it looks like I need a separate class for
> each subscriber.  Inside that class, I'd like to have a standard set
> of queues to prioritize each customer's slice of bandwidth by port
> (typical three band stuff:  interactive, web,bulk).
>   So assuming I've got three customers:
> 10.0.0.10 gets256kb/s
> 10.0.0.11 gets 386 kb/s
> 10.0.0.12 gets 256kb/s
>  I think I'd have a tree like this:
> 
>                         CLASS 10 (256kb/s) (inner classes prioritize)
> ROOT -----        CLASS 11(384kb/s)                 "
>                         CLASS 12 (256kb/s)                "
> plus three filters to direct iptables-marked traffic to the
> appropriate queue and three iptables entries to mark the traffic
> appropriately.
> 
> The problem is, I'm simply not smart enough to actually IMPLEMENT
> this.  I tried setting up the HTB classes, and when I added the 5th
> one (class 15 in the little diagram above), my interactivity to the
> router went to near 1000ms RTT (I was ssh'ed into it).  I started
> getting calls from my customers immediately saying something was
> 'wrong with the internet'.  I'm not sure what happened, since I hadn't
> installed any filters or anything.
> 
> Anyway, I'd dearly appreciate some help on this.  Surely this is a nut
> that someone has already cracked, but for the life of me all I can
> find on the internet are how-to shape your own outbound traffic to
> your ISP so your P2P traffic (the BANE of ISPs) doesn't interfere with
> your Doom3 deathmatch.
> 
> Thanks in advance,
> John Gorkos
> _______________________________________________
> LARTC mailing list
> LARTC@mailman.ds9a.nl
> http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc


-- 
Brian Carrig
Research Assistant
Department of Computing & Networking
Institute of Technology, Carlow
Tel. No.: +353 59 9176314
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Too stupid to figure out shaping

[Andreas Klauer]

On Wednesday 27 April 2005 17:35, John Gorkos wrote:
> Anyway, I'd dearly appreciate some help on this.  Surely this is a nut
> that someone has already cracked, but for the life of me all I can find 
> on the internet are how-to shape your own outbound traffic to your ISP

I may be really close to this one. Although I'm not an ISP and don't have 
customers per se, I do have flatmates and a router, and I'm pretty much 
using the one class per user approach you're describing.

Although I know of someone who is using a modified version of this script 
in a wireless network, it's developed with 'home router' (especially my 
own router) and 'fair sharing' in mind. So it has many downsides for you; 
it kicks out your firewall, works only with static IPs, and capping users 
to 256/368/512kbit will only affect ceil rate.

On the bright side, as long as you have one device for internet, one device 
for customers, and your customers have static IPs on this device, it might 
work out of the box for you, and even if it doesn't, it comes with some 
documentation and structure that is supposed to make it easy to 
understand, extract ideas and modify. And from what I read in the feedback 
I get, most people use it this way. ;-)

You can find my script here: http://www.metamorpher.de/fairnat

HTH
Andreas
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Too stupid to figure out shaping

[John Gorkos]

On Wednesday 27 April 2005 13:42, Andreas Klauer wrote:
> On Wednesday 27 April 2005 17:35, John Gorkos wrote:
> > Anyway, I'd dearly appreciate some help on this.  Surely this is a nut
> > that someone has already cracked, but for the life of me all I can find
> > on the internet are how-to shape your own outbound traffic to your ISP
>
> I may be really close to this one. Although I'm not an ISP and don't have
> customers per se, I do have flatmates and a router, and I'm pretty much
> using the one class per user approach you're describing.
>
> Although I know of someone who is using a modified version of this script
> in a wireless network, it's developed with 'home router' (especially my
> own router) and 'fair sharing' in mind. So it has many downsides for you;
> it kicks out your firewall, works only with static IPs, and capping users
> to 256/368/512kbit will only affect ceil rate.
>
> On the bright side, as long as you have one device for internet, one device
> for customers, and your customers have static IPs on this device, it might
> work out of the box for you, and even if it doesn't, it comes with some
> documentation and structure that is supposed to make it easy to
> understand, extract ideas and modify. And from what I read in the feedback
> I get, most people use it this way. ;-)
>
> You can find my script here: http://www.metamorpher.de/fairnat
>
> HTH
> Andreas

This is remarkably close to what I'm looking for.  The grin on my face should 
be visible from Germany.  I have a question about the following section from 
the config file:
# --- Clients: ---

# Specify the clients for which we do Masquerading and Shaping.
# The script assumes that all clients have static IPs in the
# same subnet as your LAN device.
#
# Example: If the IP of DEV_LAN is 192.168.100.42, the line above
#          means that 192.168.100.2, 192.168.100.5, etc., are
#          the IPs of your clients.
#          6:23:25 is a group of 3 IPs that all belong to the same user.
#          Use this notation if a single person has more than one machine
#          /IP connected to the router.
#
# New: You can also specify a custom ceil rate per user.
#      Syntax:  <user>@<down_ceil>[|<up_ceil>]
#      However, ceil has to be bigger than the guaranteed rate,
#      otherwise you will get weird results.
USERS="2 5 6:23:25 183@1mbit|100kbit"

How do I include multiple lines of USERS= here.  For example, users with IP 
addresss 11,12,13,14,15 are all 256kb/s users, 16,17,18,19 are 384kb/s users, 
and 20-30 are back to 256kb/s users.  Is this acceptable:
USERS="\
11 12 13 14 15 183@256kbit/256kbit\
16 17 18 19 184@384kbit/384kbit\
20 21 22 23 24 25 26 27 28 29 30 185@256kbit/256kbit"
?
Also, what does the 183 (184,185 in my example) signify?

Thanks tons!
John Gorkos
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] Too stupid to figure out shaping

[Andreas Klauer]

On Thursday 28 April 2005 00:19, you wrote:
> #      Syntax:  <user>@<down_ceil>[|<up_ceil>]
> How do I include multiple lines of USERS= here.

Maybe the example is bad. USERS does not have lines.

> Also, what does the 183 (184,185 in my example) signify?

The same as 20, 21, 22, etc. in your example. That's one user. You'll have 
to specify ceil rates for all users you want to be limited that way. Also 
notice that it's | instead of /.

For example, 20@512kbit|256kbit translates to: 
User 20 will be ceiled at 512kbit download and 256kbit upload.

The number 20 will be expanded to the IP of that user in your subnet, so if 
your subnet is 192.168.100.x, the IP will be 192.168.100.20. You can also 
group multiple IPs together, so 20:21:22@256kbit|256kbit would be similar 
to your old setup, throwing traffic of all IPs together in the same class 
and let them battle it out.

So if I interpret your example right, it should look like this:

USERS="11@256kbit|256kbit 12@256kbit|256kbit 13@256kbit|256kbit 
       14@256kbit|256kbit 15@256kbit|256kbit 16@386kbit|386kbit 
       17@386kbit|386kbit 18@386kbit|386kbit 19@386kbit|386kbit 
       20@256kbit|256kbit 21@256kbit|256kbit 22@256kbit|256kbit 
       23@256kbit|256kbit 24@256kbit|256kbit 25@256kbit|256kbit
       26@256kbit|256kbit 27@256kbit|256kbit 28@256kbit|256kbit
       29@256kbit|256kbit 30@256kbit|256kbit"

Yeah, it's very ugly. That's because when I started you didn't even have 
the possibility to specify ceil rates like this (the original idea was to 
give every flatmate an equal sized piece of the cake). USERS="1 2 3 4 5" 
is all you need for that...

HTH
Andreas
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc