Stopping greedy http clients!

Stopping greedy http clients!

[J and T]

Is there a way to set a "speed limit" on tcp port 80 requests? Here's my 
problem:

I have a very popular site with hundreds of pages updated dynamically every 
4 hours. Since high-speed internet access has become more popular, I have 
found more and more visitors using robots to download all pages in one quick 
multi-threaded connection. When you have a 1,000 people downloading 100 
pages all at once you end up with a problem. I can't just block them by 
their IP because their IPs are dynamic.

I've tried mod_throttle (by Apache), but because my site is very popular I 
have to keep a huge IP list in memory so this is super memory consuming. Not 
only that, but because the list is so huge response time on the server is 
slowed considerably.

My wish is to have a speed limit. So if a user is detected of downloading x 
number of documents in x amount of time, they are DROPed. Then after a "wait 
period" they are given permission again. Experimenting a little with this 
has caused problems. The problem is that a Web page could easily have 30 
small images so one http requeset for page a could result in 31 requests.

I don't want to set a bandwidth limit because that will only reject users 
who have a legit reason to be on the site. I'm only interested in 
temporarily blocking greedy clients consuming all my resources. Right now 
I'm on a multiple T-1 so it's not bandwidth that is the problem. The problem 
is greedy clients hitting me so hard that requests are slowed down for all 
visitors.

I know yahoo, ebay, cnn and usatoday have all experienced this problem, but 
I don't know how they solved it.

Anyone have a suggestion on how to go about doing this?

Thanks


_________________________________________________________________
Join the world’s largest e-mail service with MSN Hotmail. 
http://www.hotmail.com

Re: Stopping greedy http clients!

[david]

On Mon, Sep 23, 2002 at 05:52:50PM -0700, J and T wrote:
> Is there a way to set a "speed limit" on tcp port 80 requests? Here's my 
> problem:
> 
> I have a very popular site with hundreds of pages updated dynamically every 
> 4 hours. Since high-speed internet access has become more popular, I have 
> found more and more visitors using robots to download all pages in one 
> quick multi-threaded connection. When you have a 1,000 people downloading 
> 100 pages all at once you end up with a problem. I can't just block them by 
> their IP because their IPs are dynamic.

I would take a look at iplimit from patch-o-matic, which "allows you to
restrict the number of parallel TCP connections to a server per client IP
address (or address block)". It will not block the access to your site, but
you can use it to limit the number of parallel downloads the clients can
use.

- David

RE: Stopping greedy http clients!

[Rob Sterenborg]

[-- Attachment #1: Type: text/plain, Size: 452 bytes --]

> > Is there a way to set a "speed limit" on tcp port 80 

> I would take a look at iplimit from patch-o-matic, which 
> "allows you to
> restrict the number of parallel TCP connections to a server 
> per client IP
> address (or address block)". It will not block the access to 

Or use something (not netfilter related) like CBQ or HTB which implement
bandwidth throttling.
I found CBQ more difficult to setup than HTB, but the choice is yours.


Rob

[-- Attachment #2: Type: text/html, Size: 1020 bytes --]

RE: Stopping greedy http clients!

[Maciej Soltysiak]

> Or use something (not netfilter related) like CBQ or HTB which implement
> bandwidth throttling.
> I found CBQ more difficult to setup than HTB, but the choice is yours.
There is a 3rd party module to apache, that allows to setup bandwidth and
concurrent connections, without making classes for traffic via tc.
I do not remember the nam right now, but it exists :)

Regards,
Maciej Soltysiak

RE: Stopping greedy http clients!

[Matt Ryanczak]

[-- Attachment #1: Type: text/plain, Size: 694 bytes --]

Checkout mod_bandwidth for apache @ 
http://www.snert.com/Software/mod_throttle/

On Tue, 2002-09-24 at 10:41, Maciej Soltysiak wrote:
> > Or use something (not netfilter related) like CBQ or HTB which implement
> > bandwidth throttling.
> > I found CBQ more difficult to setup than HTB, but the choice is yours.
> There is a 3rd party module to apache, that allows to setup bandwidth and
> concurrent connections, without making classes for traffic via tc.
> I do not remember the nam right now, but it exists :)
> 
> Regards,
> Maciej Soltysiak
-- 
Matt Ryanczak
System Administrator
American Registy of Internet Numbers
ryanczakm@arin.net
(o)703.227.9869
(m)703.926.2595

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 232 bytes --]

Re: Stopping greedy http clients!

[J and T]

[-- Attachment #1: Type: text/html, Size: 5571 bytes --]