[LARTC] urgent question about tcng!

[LARTC] urgent question about tcng!

[Thomas Mandl]

Hello List,
I'm new to QoS/tcng/HTB and friends, so please forgive me if my question
might be silly...
After having read lots of HowTo documents I'm totally confused...

The Challenge:
=======
I'll have to deploy several "mirror" download servers (Linux) which must be
able to handle a huge number of HTTP download requests (about 10k to 20k
unicast requests per server within 120 min.) which will hit my servers . The
mirror servers will be connected with 100MBit to the Internet backbone. Each
"client" will download between 7MB up to 30MB per request. The download
mirror servers will be directly attached to the Internet (no FW or router
between, which could handle QoS for me). Of course will these download
mirrors be specially hardened :-)

The Idea:
====I can not handle all the traffic at once without QoS, so I thought it should
be possible to establish a traffic control mechanism to guarantee bandwith
for each client download (e.g. 256kbps CIR up to 512kbps ceil). This should
allow me, to "serve" all clients within 120 min, before a new download wave
will hit my servers. If the available bandwith for HTTP traffic is used,
clients shall not be able to start a download, until some bandwith is freed
by another client who finished a download. 

Note: I would rather go with "tcng" than with the old "tc".

The Problem:
======
While playing with tcng and HTB I managed to shape traffic (to limit the
total bandwith for HTTP traffic to e.g. 50Mbps) but I did not manage to
write a tcng configuration which guarantees a certain bandwith per client
request. I must be able to assign (and guarantee) a defined bandwith to a
client request, to make the downloads more predictable. My experiments so
far only allowed me to limit e.g. HTTP traffic, which then was equally
distributed to the number of clients, but this led to the usual "traffic
jam" when too many clients initiated a simultanes download.

Can anybody please assist me, or does anybody have a working "tcng" config
file to do the job. Additional readings/comments/links (beside those listed
in the LDB/Linux Traffic Control Howtos are also welcome.)

Many thanks in advance and
best regards from Austria

Thomas

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] urgent question about tcng!

[Sylvain BERTRAND]

On Mer 27 avril 2005 18:11, Thomas Mandl a écrit :
> Hello List,
> I'm new to QoS/tcng/HTB and friends, so please forgive me if my question
might be silly...
> After having read lots of HowTo documents I'm totally confused...
>
> The Challenge:
> =======
> I'll have to deploy several "mirror" download servers (Linux) which must be
> able to handle a huge number of HTTP download requests (about 10k to 20k
unicast requests per server within 120 min.) which will hit my servers .
The
> mirror servers will be connected with 100MBit to the Internet backbone.
Each
> "client" will download between 7MB up to 30MB per request. The download
mirror servers will be directly attached to the Internet (no FW or
router between, which could handle QoS for me). Of course will these
download mirrors be specially hardened :-)
>
> The Idea:
> ====> I can not handle all the traffic at once without QoS, so I thought it
should
> be possible to establish a traffic control mechanism to guarantee
bandwith for each client download (e.g. 256kbps CIR up to 512kbps ceil).
This should
> allow me, to "serve" all clients within 120 min, before a new download wave
> will hit my servers. If the available bandwith for HTTP traffic is used,
clients shall not be able to start a download, until some bandwith is
freed
> by another client who finished a download.
>
> Note: I would rather go with "tcng" than with the old "tc".
>
> The Problem:
> ======
> While playing with tcng and HTB I managed to shape traffic (to limit the
total bandwith for HTTP traffic to e.g. 50Mbps) but I did not manage to
write a tcng configuration which guarantees a certain bandwith per
client request. I must be able to assign (and guarantee) a defined
bandwith to a client request, to make the downloads more predictable. My
experiments so far only allowed me to limit e.g. HTTP traffic, which
then was equally distributed to the number of clients, but this led to
the usual "traffic jam" when too many clients initiated a simultanes
download.
>
> Can anybody please assist me, or does anybody have a working "tcng"
config file to do the job. Additional readings/comments/links (beside
those listed
> in the LDB/Linux Traffic Control Howtos are also welcome.)
>
> Many thanks in advance and
> best regards from Austria
>
> Thomas
>

I think you might want to check Apache's mod_bandwidth / mod_throttle
instead of QoS.

Regards,

Sylvain



_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] urgent question about tcng!

[Jason Boxman]

On Wednesday 27 April 2005 12:17, Sylvain BERTRAND wrote:
<snip>
> I think you might want to check Apache's mod_bandwidth / mod_throttle
> instead of QoS.

There's also bw_mod if you're running Apache2, which for me is working fine so 
far.

http://www.ivn.cl/apache/

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: [LARTC] urgent question about tcng!

[Andy Furniss]

Thomas Mandl wrote:
> Hello List,
> I'm new to QoS/tcng/HTB and friends, so please forgive me if my question
> might be silly...
> After having read lots of HowTo documents I'm totally confused...
> 
> The Challenge:
> =======
> I'll have to deploy several "mirror" download servers (Linux) which must be
> able to handle a huge number of HTTP download requests (about 10k to 20k
> unicast requests per server within 120 min.) which will hit my servers . The
> mirror servers will be connected with 100MBit to the Internet backbone. Each
> "client" will download between 7MB up to 30MB per request. The download
> mirror servers will be directly attached to the Internet (no FW or router
> between, which could handle QoS for me). Of course will these download
> mirrors be specially hardened :-)
> 
> The Idea:
> ====> I can not handle all the traffic at once without QoS, so I thought it should
> be possible to establish a traffic control mechanism to guarantee bandwith
> for each client download (e.g. 256kbps CIR up to 512kbps ceil). This should
> allow me, to "serve" all clients within 120 min, before a new download wave
> will hit my servers. If the available bandwith for HTTP traffic is used,
> clients shall not be able to start a download, until some bandwith is freed
> by another client who finished a download. 
> 
> Note: I would rather go with "tcng" than with the old "tc".
> 
> The Problem:
> ======
> While playing with tcng and HTB I managed to shape traffic (to limit the
> total bandwith for HTTP traffic to e.g. 50Mbps) but I did not manage to
> write a tcng configuration which guarantees a certain bandwith per client
> request. I must be able to assign (and guarantee) a defined bandwith to a
> client request, to make the downloads more predictable. My experiments so
> far only allowed me to limit e.g. HTTP traffic, which then was equally
> distributed to the number of clients, but this led to the usual "traffic
> jam" when too many clients initiated a simultanes download.
> 
> Can anybody please assist me, or does anybody have a working "tcng" config
> file to do the job. Additional readings/comments/links (beside those listed
> in the LDB/Linux Traffic Control Howtos are also welcome.)

Wang Jian posted a perflow queue recently -

http://mailman.ds9a.nl/pipermail/lartc/2005q2/015381.html

It is the sort of thing you need if you can't do it with apache as 
recommended.

Andy.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

AW: [LARTC] urgent question about tcng!

[Thomas Mandl]

Hello Everybody,

I forgot to mention, that my HTTP traffic will not be handled by an apache
web server, and the HTTP protocol can be replaced in the near future by a
different protocol. Thus apache's mod_bandwith isn't an option for me right
now.

I'm not sure how Wang Jian's post can help me. I thought the existing Linux
2.6.11 kernel and tcng already can do what I need? All I have to do is write
a proper tcng config file...

But the proper tcng file is the challenge...

regards
Thomas
-----Ursprüngliche Nachricht-----
Von: Andy Furniss [mailto:andy.furniss@dsl.pipex.com] 
Gesendet: Mittwoch, 27. April 2005 23:23
An: mandl.t@ikarus.at
Cc: LARTC
Betreff: Re: [LARTC] urgent question about tcng!

Thomas Mandl wrote:
> Hello List,
> I'm new to QoS/tcng/HTB and friends, so please forgive me if my 
> question might be silly...
> After having read lots of HowTo documents I'm totally confused...
> 
> The Challenge:
> =======
> I'll have to deploy several "mirror" download servers (Linux) which 
> must be able to handle a huge number of HTTP download requests (about 
> 10k to 20k unicast requests per server within 120 min.) which will hit 
> my servers . The mirror servers will be connected with 100MBit to the 
> Internet backbone. Each "client" will download between 7MB up to 30MB 
> per request. The download mirror servers will be directly attached to 
> the Internet (no FW or router between, which could handle QoS for me). 
> Of course will these download mirrors be specially hardened :-)
[...snip...]
> 
> Can anybody please assist me, or does anybody have a working "tcng" 
> config file to do the job. Additional readings/comments/links (beside 
> those listed in the LDB/Linux Traffic Control Howtos are also 
> welcome.)

Wang Jian posted a perflow queue recently -

http://mailman.ds9a.nl/pipermail/lartc/2005q2/015381.html

It is the sort of thing you need if you can't do it with apache as
recommended.

Andy.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: AW: [LARTC] urgent question about tcng!

[Andy Furniss]

Thomas Mandl wrote:
> Hello Everybody,
> 
> I forgot to mention, that my HTTP traffic will not be handled by an apache
> web server, and the HTTP protocol can be replaced in the near future by a
> different protocol. Thus apache's mod_bandwith isn't an option for me right
> now.
> 
> I'm not sure how Wang Jian's post can help me. I thought the existing Linux
> 2.6.11 kernel and tcng already can do what I need? All I have to do is write
> a proper tcng config file...
> 
> But the proper tcng file is the challenge...

I don't know tcng, but the reason I suggested perflow is that you want 
each flow to have a ceil - unless you make a class and rule to match 
each flow I can't see how you can do this. Also iptables could limit the 
number of connections - tc can't, perflow can.

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: AW: [LARTC] urgent question about tcng!

[Andy Furniss]

Andy Furniss wrote:

> 
> I don't know tcng, but the reason I suggested perflow is that you want 
> each flow to have a ceil - unless you make a class and rule to match 
> each flow I can't see how you can do this. Also iptables could limit the 
> number of connections - tc can't, perflow can.

Forgot to say you can also use iptables to limit rate per connection.

Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

AW: AW: [LARTC] urgent question about tcng!

[Thomas Mandl]

Sorry for the stupid question, but how would I rate limit connections using
iptables?

regards
Thomas 
-----Ursprüngliche Nachricht-----
Von: Andy Furniss [mailto:andy.furniss@dsl.pipex.com] 
Gesendet: Dienstag, 03. Mai 2005 16:43
An: Andy Furniss
Cc: mandl.t@ikarus.at; 'LARTC'
Betreff: Re: AW: [LARTC] urgent question about tcng!

Andy Furniss wrote:

> 
> I don't know tcng, but the reason I suggested perflow is that you want 
> each flow to have a ceil - unless you make a class and rule to match 
> each flow I can't see how you can do this. Also iptables could limit 
> the number of connections - tc can't, perflow can.

Forgot to say you can also use iptables to limit rate per connection.

Andy.

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc

Re: AW: AW: [LARTC] urgent question about tcng!

[Andy Furniss]

Thomas Mandl wrote:
> Sorry for the stupid question, but how would I rate limit connections using
> iptables?
> 

I was thinking of the patch-o-matic-ng patches connlimit and connrate, 
though I've never used either and they may or may not be expensive for 
many connections compared to perflow.

Connrate lets you mark packets if they are above limits - you could then 
drop those later in a filter table or with a TC filter/queue.

> regards
> Thomas 
> -----Ursprüngliche Nachricht-----
> Von: Andy Furniss [mailto:andy.furniss@dsl.pipex.com] 
> Gesendet: Dienstag, 03. Mai 2005 16:43
> An: Andy Furniss
> Cc: mandl.t@ikarus.at; 'LARTC'
> Betreff: Re: AW: [LARTC] urgent question about tcng!
> 
> Andy Furniss wrote:
> 
> 
>>I don't know tcng, but the reason I suggested perflow is that you want 
>>each flow to have a ceil - unless you make a class and rule to match 
>>each flow I can't see how you can do this. Also iptables could limit 
>>the number of connections - tc can't, perflow can.
> 
> 
> Forgot to say you can also use iptables to limit rate per connection.
> 
> Andy.
> 
> 

_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc