* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
@ 2005-05-26 13:49 ` Justin Schoeman
2005-05-26 14:00 ` Marcin Kałuża
` (12 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Justin Schoeman @ 2005-05-26 13:49 UTC (permalink / raw)
To: lartc
I haven't tried this myself, but I see l7filter (l7-filter.sf.net) has
patterns for matching cache hits and misses... This may just work for
your application?
-justin
Peter Kaagman wrote:
> Hi list...
>
> I work for a school in the netherlands with a 2mbit Internet uplink and
> about 3800 eager student who want to play games on the Internet using
> one of our 800 workstations.
>
> Problem was that those game playing students are concentrated in 2 of
> our 6 physical locations... and they consumed the bandwidth which the
> other location would like to use for educational purposes.
>
> The thing we did first was use squid... with success. The hit ratio on
> data transfer is 25-30%... "free" bandwidth.
>
> Today I took the plunge and started to use HTB traffic shaping... and
> (to my surprise) I got it going without much troubles.
>
> The setup I have chosen first divides the load over two classes:
> - one for Internet rate 2mbit and a 2mbit ceil
> - a second for our DMZ rate 98mbit and a 100mbit ceil
>
> Next I sub-classed the Internet bucket into 6 classes each with a
> 333kbit rate and a 2mbit ceil.
>
> This has had the effect that my DMZ can be accessed at full speed while
> they fairly share the Internet uplink.
>
> And the way it looks now it works :D
> Hail to all those people who wrote those fine docs _o_
>
> This is enough reason to address this list... just to say "Thank you!",
> but there is more.
>
> At the moment I do not max out my Internet link... reason for this is I
> guess the squid proxy...
> The way it works now is that I have 2 types of filters in effect:
> - The DMZ: all packages with a src ip from my DMZ go to the big 98/100
> bucket.
> - The Internet: all packages with a dst ip in one of our 6 networks
> gets placed in one of the 6 333/2000 buckets.
>
> But there is of course a src of packages I do not catch this way... and
> these are the squid cache hits. Because I filter on destination the cache
> hits get treated the same as cache misses. But cache hits are in effect
> local traffic... they do not originate from the Internet.
>
> So here (finally) the question..
> Is there a way to identify cache hits from misses?
>
> I took a look at the advanced filtering chapter of course, but am
> really dazzled by that (and I thought I understood TCP/IP a bit ;)).
>
> Some further info that would perhaps help is that squid is run as a
> transparant proxy on the router/firewall.
>
> regards
>
> Peter Kaagman
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
2005-05-26 13:49 ` Justin Schoeman
@ 2005-05-26 14:00 ` Marcin Kałuża
2005-05-26 14:04 ` Evgeni Gechev
` (11 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Marcin Kałuża @ 2005-05-26 14:00 UTC (permalink / raw)
To: lartc
One of the ways is to use tos field, but I think you need to patch squid for
it. Brief google search gave this:
http://www.it-academy.bg/zph/
I've never used it though so I'm not sure if I can be of any more help on it.
Hope this helps
Martin
On Thursday 26 May 2005 15:32, Peter Kaagman wrote:
> Hi list...
>
> I work for a school in the netherlands with a 2mbit Internet uplink and
> about 3800 eager student who want to play games on the Internet using
> one of our 800 workstations.
>
> Problem was that those game playing students are concentrated in 2 of
> our 6 physical locations... and they consumed the bandwidth which the
> other location would like to use for educational purposes.
>
> The thing we did first was use squid... with success. The hit ratio on
> data transfer is 25-30%... "free" bandwidth.
>
> Today I took the plunge and started to use HTB traffic shaping... and
> (to my surprise) I got it going without much troubles.
>
> The setup I have chosen first divides the load over two classes:
> - one for Internet rate 2mbit and a 2mbit ceil
> - a second for our DMZ rate 98mbit and a 100mbit ceil
>
> Next I sub-classed the Internet bucket into 6 classes each with a
> 333kbit rate and a 2mbit ceil.
>
> This has had the effect that my DMZ can be accessed at full speed while
> they fairly share the Internet uplink.
>
> And the way it looks now it works :D
> Hail to all those people who wrote those fine docs _o_
>
> This is enough reason to address this list... just to say "Thank you!",
> but there is more.
>
> At the moment I do not max out my Internet link... reason for this is I
> guess the squid proxy...
> The way it works now is that I have 2 types of filters in effect:
> - The DMZ: all packages with a src ip from my DMZ go to the big 98/100
> bucket.
> - The Internet: all packages with a dst ip in one of our 6 networks
> gets placed in one of the 6 333/2000 buckets.
>
> But there is of course a src of packages I do not catch this way... and
> these are the squid cache hits. Because I filter on destination the cache
> hits get treated the same as cache misses. But cache hits are in effect
> local traffic... they do not originate from the Internet.
>
> So here (finally) the question..
> Is there a way to identify cache hits from misses?
>
> I took a look at the advanced filtering chapter of course, but am
> really dazzled by that (and I thought I understood TCP/IP a bit ;)).
>
> Some further info that would perhaps help is that squid is run as a
> transparant proxy on the router/firewall.
>
> regards
>
> Peter Kaagman
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
2005-05-26 13:49 ` Justin Schoeman
2005-05-26 14:00 ` Marcin Kałuża
@ 2005-05-26 14:04 ` Evgeni Gechev
2005-05-26 14:05 ` Daniel Lupescu
` (10 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Evgeni Gechev @ 2005-05-26 14:04 UTC (permalink / raw)
To: lartc
Peter Kaagman wrote:
>Hi list...
>
>I work for a school in the netherlands with a 2mbit Internet uplink and
>about 3800 eager student who want to play games on the Internet using
>one of our 800 workstations.
>
>Problem was that those game playing students are concentrated in 2 of
>our 6 physical locations... and they consumed the bandwidth which the
>other location would like to use for educational purposes.
>
>The thing we did first was use squid... with success. The hit ratio on
>data transfer is 25-30%... "free" bandwidth.
>
>Today I took the plunge and started to use HTB traffic shaping... and
>(to my surprise) I got it going without much troubles.
>
>The setup I have chosen first divides the load over two classes:
>- one for Internet rate 2mbit and a 2mbit ceil
>- a second for our DMZ rate 98mbit and a 100mbit ceil
>
>Next I sub-classed the Internet bucket into 6 classes each with a
>333kbit rate and a 2mbit ceil.
>
>This has had the effect that my DMZ can be accessed at full speed while
>they fairly share the Internet uplink.
>
>And the way it looks now it works :D
>Hail to all those people who wrote those fine docs _o_
>
>This is enough reason to address this list... just to say "Thank you!",
>but there is more.
>
>At the moment I do not max out my Internet link... reason for this is I
>guess the squid proxy...
>The way it works now is that I have 2 types of filters in effect:
>- The DMZ: all packages with a src ip from my DMZ go to the big 98/100
> bucket.
>- The Internet: all packages with a dst ip in one of our 6 networks
> gets placed in one of the 6 333/2000 buckets.
>
>But there is of course a src of packages I do not catch this way... and
>these are the squid cache hits. Because I filter on destination the cache
>hits get treated the same as cache misses. But cache hits are in effect
>local traffic... they do not originate from the Internet.
>
>So here (finally) the question..
>Is there a way to identify cache hits from misses?
>
>I took a look at the advanced filtering chapter of course, but am
>really dazzled by that (and I thought I understood TCP/IP a bit ;)).
>
>Some further info that would perhaps help is that squid is run as a
>transparant proxy on the router/firewall.
>
>regards
>
>Peter Kaagman
>
>
http://www.it-academy.bg/zph/
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (2 preceding siblings ...)
2005-05-26 14:04 ` Evgeni Gechev
@ 2005-05-26 14:05 ` Daniel Lupescu
2005-05-26 14:33 ` Peter Surda
` (9 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Daniel Lupescu @ 2005-05-26 14:05 UTC (permalink / raw)
To: lartc
Peter Kaagman wrote:
> Hi list...
> So here (finally) the question..
> Is there a way to identify cache hits from misses?
there is a patch for squid available at http://www.it-academy.bg/zph/
> Some further info that would perhaps help is that squid is run as a
> transparant proxy on the router/firewall.
>
> regards
>
> Peter Kaagman
--
dlupescu
--
This message was scanned for spam and viruses by BitDefender.
For more information please visit http://www.bitdefender.com/
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (3 preceding siblings ...)
2005-05-26 14:05 ` Daniel Lupescu
@ 2005-05-26 14:33 ` Peter Surda
2005-05-26 17:49 ` Pan'ko Alexander
` (8 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Peter Surda @ 2005-05-26 14:33 UTC (permalink / raw)
To: lartc
On Thu, 26 May 2005 15:32:42 +0200 Peter Kaagman <p.kaagman@atlascollege.nl>
wrote:
>Problem was that those game playing students are concentrated in 2 of
>our 6 physical locations... and they consumed the bandwidth which the
>other location would like to use for educational purposes.
Actually, according to my experience, online gaming requires only a little
bandwidth (~3-5kB/s) but sustained over a longer period. Furthermore, if the
latency jumps above ~200ms it becomes less playable, and above about 500ms it's
practically useless, so noone will be able to play anyway. IMHO your bandwidth
is consumed by P2P applications or worms, which have a much more serious effect
on this.
I have a linux distribution (Route Hat) optimised for this type of application
(many unrelated computers sharing the same line). It may help you and even if
not directly, you can take some hints from the scripts. In fact several
dormitories already use it to great satisfaction.
>Peter Kaagman
Yours sincerely,
Peter
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (4 preceding siblings ...)
2005-05-26 14:33 ` Peter Surda
@ 2005-05-26 17:49 ` Pan'ko Alexander
2005-05-26 18:56 ` Andy Furniss
` (7 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Pan'ko Alexander @ 2005-05-26 17:49 UTC (permalink / raw)
To: lartc
On Thu, 26 May 2005 15:32:42 +0200
Peter Kaagman <p.kaagman@atlascollege.nl> wrote:
> So here (finally) the question..
> Is there a way to identify cache hits from misses?
Maybe I do not understend question, but I think it is very simple.
There is option tcp_outgoing_address. Note it mean replacing source address.
Then each user machine have it's owne address.
For this aim I added to dummy (may be any other) interface the addresses of another subnet.
squid-2.5.STABLE9-1.100.6asp
It's working...
But not working IMQ, that I need too. I do not understand strange intension to use only PREROUTING and POSTROUTING.
If you will use IMQ you will need AB instead of default BA NAT
--
С наилучшими пожеланиями, Панько Александр.
With best regards, Pan'ko Alexander.
pankoAA@yandex.ru
http://interdon.net/~panko/
ICQ 231647363
XMMS playing nothing :-)
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (5 preceding siblings ...)
2005-05-26 17:49 ` Pan'ko Alexander
@ 2005-05-26 18:56 ` Andy Furniss
2005-05-26 19:13 ` Pan'ko Alexander
` (6 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Andy Furniss @ 2005-05-26 18:56 UTC (permalink / raw)
To: lartc
Peter Kaagman wrote:
> But there is of course a src of packages I do not catch this way... and
> these are the squid cache hits. Because I filter on destination the cache
> hits get treated the same as cache misses. But cache hits are in effect
> local traffic... they do not originate from the Internet.
If squid is running on the same machine as your htb rules then (I think)
the only way you can shape incoming traffic from the internet properly
is to use imq.
I have not used squid - so may be wrong, but the patches will let you
classify hits so they can be let through at lan speed. But what about
misses - I assume that squid will connect to the internet and fetch the
data unlimited even if they then get served to the lan at restricted speed.
Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (6 preceding siblings ...)
2005-05-26 18:56 ` Andy Furniss
@ 2005-05-26 19:13 ` Pan'ko Alexander
2005-05-26 19:41 ` Andy Furniss
` (5 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Pan'ko Alexander @ 2005-05-26 19:13 UTC (permalink / raw)
To: lartc
On Thu, 26 May 2005 19:56:09 +0100
Andy Furniss <andy.furniss@dsl.pipex.com> wrote:
> Peter Kaagman wrote:
>
> > But there is of course a src of packages I do not catch this way... and
> > these are the squid cache hits. Because I filter on destination the cache
> > hits get treated the same as cache misses. But cache hits are in effect
> > local traffic... they do not originate from the Internet.
>
> If squid is running on the same machine as your htb rules then (I think)
> the only way you can shape incoming traffic from the internet properly
> is to use imq.
I think IMQ needed only if there are not one interface to shape.
>
> I have not used squid - so may be wrong, but the patches will let you
> classify hits so they can be let through at lan speed. But what about
> misses - I assume that squid will connect to the internet and fetch the
> data unlimited even if they then get served to the lan at restricted speed.
>
Inet <-----(one for all src)NAT<--(MISSES)---(src 192.168.90.0/28 dst 'real Inet IP')Squid<---(HITS+MISSES)---hosts
Inet ----->(one for all dst)NAT--(MISSES)--->(dst 192.168.90.0/28 src 'real Inet IP')Squid---(HITS+MISSES)--->hosts
The last not fully right... But clenly illustrates the idea.
You can simply shape the MISSES on one interface...
--
С наилучшими пожеланиями, Панько Александр.
With best regards, Pan'ko Alexander.
pankoAA@yandex.ru
http://interdon.net/~panko/
ICQ 231647363
XMMS playing nothing :-)
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (7 preceding siblings ...)
2005-05-26 19:13 ` Pan'ko Alexander
@ 2005-05-26 19:41 ` Andy Furniss
2005-05-26 19:47 ` Peter Kaagman
` (4 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Andy Furniss @ 2005-05-26 19:41 UTC (permalink / raw)
To: lartc
Pan'ko Alexander wrote:
> On Thu, 26 May 2005 19:56:09 +0100
> Andy Furniss <andy.furniss@dsl.pipex.com> wrote:
>
>
>>Peter Kaagman wrote:
>>
>>
>>>But there is of course a src of packages I do not catch this way... and
>>>these are the squid cache hits. Because I filter on destination the cache
>>>hits get treated the same as cache misses. But cache hits are in effect
>>>local traffic... they do not originate from the Internet.
>>
>>If squid is running on the same machine as your htb rules then (I think)
>>the only way you can shape incoming traffic from the internet properly
>>is to use imq.
>
>
> I think IMQ needed only if there are not one interface to shape.
>
>
>>I have not used squid - so may be wrong, but the patches will let you
>>classify hits so they can be let through at lan speed. But what about
>>misses - I assume that squid will connect to the internet and fetch the
>>data unlimited even if they then get served to the lan at restricted speed.
>>
>
>
>
> Inet <-----(one for all src)NAT<--(MISSES)---(src 192.168.90.0/28 dst 'real Inet IP')Squid<---(HITS+MISSES)---hosts
>
> Inet ----->(one for all dst)NAT--(MISSES)--->(dst 192.168.90.0/28 src 'real Inet IP')Squid---(HITS+MISSES)--->hosts
>
> The last not fully right... But clenly illustrates the idea.
>
> You can simply shape the MISSES on one interface...
>
Well remember I don't use squid so don't really know, but I imagine that
all lan connections on the relevant ports go to squid and squid then
makes seperate connections to the internet if required. So all traffic
headed from the internet to squid will have the dst IP of the internet
interface even if you hook imq after (de)NAT other traffic will have
local dst addresses.
Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (8 preceding siblings ...)
2005-05-26 19:41 ` Andy Furniss
@ 2005-05-26 19:47 ` Peter Kaagman
2005-05-26 19:59 ` Peter Kaagman
` (3 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Peter Kaagman @ 2005-05-26 19:47 UTC (permalink / raw)
To: lartc
On Thu, May 26, 2005 at 07:56:09PM +0100, Andy Furniss wrote:
> Peter Kaagman wrote:
>
> >But there is of course a src of packages I do not catch this way... and
> >these are the squid cache hits. Because I filter on destination the cache
> >hits get treated the same as cache misses. But cache hits are in effect
> >local traffic... they do not originate from the Internet.
>
> If squid is running on the same machine as your htb rules then (I think)
> the only way you can shape incoming traffic from the internet properly
> is to use imq.
>
Not really sure what imq is, will have to look that up tomorrow.
> I have not used squid - so may be wrong, but the patches will let you
> classify hits so they can be let through at lan speed. But what about
> misses - I assume that squid will connect to the internet and fetch the
> data unlimited even if they then get served to the lan at restricted speed.
>
> Andy.
I think you have a point there... atm I am only shaping at eth1, which
is the LAN interface. I will still have to shape eth0, which is my
Internet interface. Not only for squid to behave, but also to give my
DMZ (and services on the LAN) a fighting change to connect when the
link is full. But after shaping eth1, with all those classes for all
the networks, shaping eth0 should be a piece of cake (I hope).
But what I saw with iptraf was that eth1 "maxed out" at 2mbit while
eth0 had a load of about 1.8mbit (in only maxed out on bursts). That
was exactly what made me wonder where that difference came from. My
best idea was that the difference was caused by the cache hits.
But I think the squid zhp patch at it-academy.bg will solve my miss/hit
problem. Taken from it's documentation it does exactly what I want.
Will try that tomorrow. Re-compiling squid is not really a problem...
build the (slackware) package myself anyway since it did not come with
the stock distro.
After that I will probably have to look in to prioritising things like
DNS, SMTP, IMAP and SSH. Although getting stuff out of our network is not
really a problem, only the download is congested.
Peter
PS
I found out that a couple of replies I made were actually private
messages... sorry about that. Most of what was said it them is repeated
in this message... except for the "Thanks for the quick reply"
--
Frisbeetarianism, n.:
The belief that when you die, your soul goes up on the roof and
gets stuck.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (9 preceding siblings ...)
2005-05-26 19:47 ` Peter Kaagman
@ 2005-05-26 19:59 ` Peter Kaagman
2005-05-27 0:09 ` Lewis Shobbrook
` (2 subsequent siblings)
13 siblings, 0 replies; 15+ messages in thread
From: Peter Kaagman @ 2005-05-26 19:59 UTC (permalink / raw)
To: lartc
On Thu, May 26, 2005 at 08:41:32PM +0100, Andy Furniss wrote:
> Well remember I don't use squid so don't really know, but I imagine that
> all lan connections on the relevant ports go to squid and squid then
> makes seperate connections to the internet if required. So all traffic
> headed from the internet to squid will have the dst IP of the internet
> interface even if you hook imq after (de)NAT other traffic will have
> local dst addresses.
What I have seen is what happens on eth1... my LAN interface.
I am able to use 2 kind of filters:
- One kind on src ip: in this filter I check for my DMZ network and put
the traffic in the LAN-speed class.
- One kind on dst ip: in which I split up the rest of the traffic up for
the 6 classes at Internet-speed
This is because traffic on eth1 is de-NATed and squid spoofs the src ip
of the original site. But again... this is just experience from testing
it and deduction. And it seems to be working ;) If I am able to set the
Tos field to a certain value for hits, I presume I can make a third
kind of filter to put these hit-packages in the LAN-speed class.
Peter
--
Reisner's Rule of Conceptual Inertia:
If you think big enough, you'll never have to do it.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (10 preceding siblings ...)
2005-05-26 19:59 ` Peter Kaagman
@ 2005-05-27 0:09 ` Lewis Shobbrook
2005-05-27 9:13 ` Konrad
2005-05-30 15:01 ` Andy Furniss
13 siblings, 0 replies; 15+ messages in thread
From: Lewis Shobbrook @ 2005-05-27 0:09 UTC (permalink / raw)
To: lartc
On Thursday 26 May 2005 11:32 pm, Peter Kaagman wrote:
Hi Peter,
> The way it works now is that I have 2 types of filters in effect:
> - The DMZ: all packages with a src ip from my DMZ go to the big 98/100
> bucket.
> - The Internet: all packages with a dst ip in one of our 6 networks
> gets placed in one of the 6 333/2000 buckets.
>
> But there is of course a src of packages I do not catch this way... and
> these are the squid cache hits. Because I filter on destination the cache
> hits get treated the same as cache misses. But cache hits are in effect
> local traffic... they do not originate from the Internet.
>
> So here (finally) the question..
> Is there a way to identify cache hits from misses?
Another approach would be to make use of the delay pools feature in squid. As
you are using a transparent proxy, you could allocate more to squid through
HTB on the internal interface for cached material and clamp squid incl. each
user/group according to an ACL on the inet interface. This way you can more
finely control the total amount of bandwidth, bandwidth per user/group etc.
is able to suck through your inet interface. You can't easily discriminate
groups using plain ol htb.
e.g. in squid.conf something like
delay_pools 2 # 2 delay pools
delay_class 1 2 # pool 1 is a class 2 pool
delay_class 2 2 # pool 2 is a class 2 pool
delay_access 1 allow admins
delay_access 1 deny all
delay_access 2 allow our_networks
delay_access 2 deny all
delay_parameters 1 -1/-1 32000/24000 48000/15000
delay_parameters 2 -1/-1 24000/15000 24000/15000
See the squid.conf doc's for a description.
Cheers,
Lewis
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (11 preceding siblings ...)
2005-05-27 0:09 ` Lewis Shobbrook
@ 2005-05-27 9:13 ` Konrad
2005-05-30 15:01 ` Andy Furniss
13 siblings, 0 replies; 15+ messages in thread
From: Konrad @ 2005-05-27 9:13 UTC (permalink / raw)
To: lartc
tc filter add dev <dev> protocol ip parent 1:0 pref 1 u32 match ip dst
192.168.0.12 match ip sport 8080 0xffff match ip tos 8 0xff flowid 1:12
#If squid miss, traffic will go to user queue.
tc filter add dev imq1 protocol ip parent 1:0 pref 2 u32 match ip src
192.168.0.22 match ip sport 8080 0xffff flowid 1:30
#this catch all traffic from source port Squid... but with priority 2
and throw in LAN class.
I'm using this patch:
--- CUT ---
diff -cr squid-2.5.STABLE3/src/client_side.c
squid-2.5.STABLE3.patched/src/client_side.c
*** squid-2.5.STABLE3/src/client_side.c 2003-05-24 13:08:41.000000000 +0200
--- squid-2.5.STABLE3.patched/src/client_side.c 2003-08-05
22:08:15.000000000 +0200
***************
*** 2005,2010 ****
--- 2005,2013 ----
/* Avoid copying to MemBuf for non-range requests */
/* Note, if we're here, then 'rep' is known to be NULL */
http->out.offset += body_size;
+ { int tos=isTcpHit(http->log_type) ? 0 : 8;
+ setsockopt(fd,SOL_IP,IP_TOS,&tos,4);
+ }
comm_write(fd, buf, size, clientWriteBodyComplete, http, NULL);
/* NULL because clientWriteBodyComplete frees it */
return;
***************
*** 2062,2067 ****
--- 2065,2073 ----
if (!http->request->range && http->request->method = METHOD_GET)
assert(check_size = size);
/* write */
+ { int tos=isTcpHit(http->log_type) ? 0 : 8;
+ setsockopt(fd,SOL_IP,IP_TOS,&tos,4);
+ }
comm_write_mbuf(fd, mb, clientWriteComplete, http);
/* if we don't do it, who will? */
memFree(buf, MEM_CLIENT_SOCK_BUF);
--- CUT ---
If squid HIT then TOS is 0, if miss then 1.
Here is the place where you can download this:
http://sed.pl/~mrk/qos/squid_hit_miss_mark.patch
Throw in to queue of LAN only traffic coming from Squid.
Upload traffic from Squid always queue in users classes.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread* Re: [LARTC] htb: HowTo identify squid cache hits
2005-05-26 13:32 [LARTC] htb: HowTo identify squid cache hits Peter Kaagman
` (12 preceding siblings ...)
2005-05-27 9:13 ` Konrad
@ 2005-05-30 15:01 ` Andy Furniss
13 siblings, 0 replies; 15+ messages in thread
From: Andy Furniss @ 2005-05-30 15:01 UTC (permalink / raw)
To: lartc
Peter Kaagman wrote:
> On Thu, May 26, 2005 at 07:56:09PM +0100, Andy Furniss wrote:
>
>>Peter Kaagman wrote:
>>
>>
>>>But there is of course a src of packages I do not catch this way... and
>>>these are the squid cache hits. Because I filter on destination the cache
>>>hits get treated the same as cache misses. But cache hits are in effect
>>>local traffic... they do not originate from the Internet.
>>
>>If squid is running on the same machine as your htb rules then (I think)
>>the only way you can shape incoming traffic from the internet properly
>>is to use imq.
>>
>
>
> Not really sure what imq is, will have to look that up tomorrow.
I think using delay pools like Lewis says could be another way.
If you don't have much traffic that needs priority over squid then you
may be able to get away with shaping on lan facing eth with the same
settings as the delay pools.
If you use imq then you won't be able to tell which user squid is
fetching the data for.
Whatever you do remember that shaping download is shaping traffic that
has already been shaped by your link - so you need to back off from the
link speed to have any chance of getting control, it still won't be
perfect if you care alot about latency.
Andy.
_______________________________________________
LARTC mailing list
LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/cgi-bin/mailman/listinfo/lartc
^ permalink raw reply [flat|nested] 15+ messages in thread