[LARTC] Parameters for the ingress qdisc?

[LARTC] Parameters for the ingress qdisc?

[Patrick Turley]

I can't find any documentation on the paramaters for the ingress qdisc.
Can someone help me?

I have a number of filters feeding into my ingress qdisc, all of which
are rate limited, but I want to place a limit on the aggregate flow as
well. I don't want to monitor the sum of the flow rates - I want to
place a hard ceiling on the total possible flow that overrides
everything else.

Any further details about the internal workings of the ingress qdisc
would also be very helpful. I've seen some stuff that suggests its
really a TBF in disguise, but I'm not sure. In particular, I'd like to
understand if the flows into the ingress qdisc can borrow from the root
and/or each other. Also, since the ingress qdisc is classless, how does
it work that each of the incoming flows are separately metered? (As I
understand it, filters only serve to guide packets into classes)

I know that the IMQ device offers a more sophisticated way to do this
but, for now, that device is not available to me.

I need answers to these specific questions, but I very much appreciate
any other information anyone would like to volunteer.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Stef Coene]

On Tuesday 05 August 2003 22:59, Patrick Turley wrote:
> I can't find any documentation on the paramaters for the ingress qdisc.
> Can someone help me?
>
> I have a number of filters feeding into my ingress qdisc, all of which
> are rate limited, but I want to place a limit on the aggregate flow as
> well. I don't want to monitor the sum of the flow rates - I want to
> place a hard ceiling on the total possible flow that overrides
> everything else.
>
> Any further details about the internal workings of the ingress qdisc
> would also be very helpful. I've seen some stuff that suggests its
> really a TBF in disguise, but I'm not sure. In particular, I'd like to
> understand if the flows into the ingress qdisc can borrow from the root
> and/or each other. Also, since the ingress qdisc is classless, how does
> it work that each of the incoming flows are separately metered? (As I
> understand it, filters only serve to guide packets into classes)
>
> I know that the IMQ device offers a more sophisticated way to do this
> but, for now, that device is not available to me.
>
> I need answers to these specific questions, but I very much appreciate
> any other information anyone would like to volunteer.
The ingress qdisc itself has no parameters.  The only thing you can do is 
using the policers.  I have a link with a patch to extend this : 
http://www.cyberus.ca/~hadi/patches/action/
Maybe this can help.

I have some more info about ingress in my mail files, but I have to sort it 
out and put it somewhere on docum.org.  But I still didn't found the the time 
to do so.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Patrick Turley]

Thank you very much for the link. That information was very helpful and
has given me a lot of insight.

Of course, the more I know, the more questions occur...

I try not to be a nuisance, so I've been doing a lot of research and I
thought I'd already found all the important web sites for Linux traffic
control. I looked again, and I still can't find anything about "filter
policers" anywhere. I didn't find any description of a command line that
even suggested such a thing was possible. Can you please point me to
some more info about this, if any exists?

The fact that the filters are metering traffic flows implies that they
have are stateful. When using filters with egress queue hierarchies, it
was my understanding that no state was needed since all they do is
direct packets into classes. It sounds like my current understanding is
quite wrong.

BTW, we are working with a stock RedHat 7.3 2.4.20-18.7 kernel, and we
are VERY reluctant to apply patches of any kind, so we're just going to
work with whatever is available in the bits we download. We're
considering moving up to 2.4.21 to get IMQ, but that doesn't appear to
be available for RedHat 7.3 yet - in fact, this may force us to RedHat
9.0 (not a bad thing, really).

On Tue, 2003-08-05 at 16:15, Stef Coene wrote:
> On Tuesday 05 August 2003 22:59, Patrick Turley wrote:
> > I can't find any documentation on the paramaters for the ingress qdisc.
> > Can someone help me?
> >
> > ...
> The ingress qdisc itself has no parameters.  The only thing you can do is 
> using the policers.  I have a link with a patch to extend this : 
> http://www.cyberus.ca/~hadi/patches/action/
> Maybe this can help.
> 
> I have some more info about ingress in my mail files, but I have to sort it 
> out and put it somewhere on docum.org.  But I still didn't found the the time 
> to do so.
> 
> Stef

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Stef Coene]

On Wednesday 06 August 2003 00:11, Patrick Turley wrote:
> Thank you very much for the link. That information was very helpful and
> has given me a lot of insight.
>
> Of course, the more I know, the more questions occur...
>
> I try not to be a nuisance, so I've been doing a lot of research and I
> thought I'd already found all the important web sites for Linux traffic
> control. I looked again, and I still can't find anything about "filter
> policers" anywhere. I didn't find any description of a command line that
> even suggested such a thing was possible. Can you please point me to
> some more info about this, if any exists?
There also some limited example scripts in the iproute2 source.

> The fact that the filters are metering traffic flows implies that they
> have are stateful. When using filters with egress queue hierarchies, it
> was my understanding that no state was needed since all they do is
> direct packets into classes. It sounds like my current understanding is
> quite wrong.
I don't think the filters are statefull.  If you use policers, you only rate 
limit the filters.  The filters itself have no idea about the traffic flows.

> BTW, we are working with a stock RedHat 7.3 2.4.20-18.7 kernel, and we
> are VERY reluctant to apply patches of any kind, so we're just going to
> work with whatever is available in the bits we download. We're
> considering moving up to 2.4.21 to get IMQ, but that doesn't appear to
> be available for RedHat 7.3 yet - in fact, this may force us to RedHat
> 9.0 (not a bad thing, really).
Why don't you download the kernel source 7.3 2.4.20-18.7 from RH and the 
config file?  So you have the same source as the kernel you trust.  If you 
trust that source, you can pacth it with what ever you want.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Patrick Turley]

> > I still can't find anything about "filter
> > policers" anywhere. I didn't find any description of a command line that
> > even suggested such a thing was possible. Can you please point me to
> > some more info about this, if any exists?
> There also some limited example scripts in the iproute2 source.

OK - I'll have a look at those scripts to see how one can associate
policers with filters.

> > The fact that the filters are metering traffic flows implies that they
> > have are stateful. When using filters with egress queue hierarchies, it
> > was my understanding that no state was needed since all they do is
> > direct packets into classes. It sounds like my current understanding is
> > quite wrong.
> I don't think the filters are statefull.  If you use policers, you only rate 
> limit the filters.  The filters itself have no idea about the traffic flows.

Then there are two further questions (the first of which I actually
should have asked before):

1) When working with egress traffic control, filters are attached to
classes. The documents I've read so far tell me that the ingress qdisc
is classless so, on the face of it, filters shouldn't be useable at all.
Please tell me which part of this I have badly misunderstood.

2) Since the filters themselves are, as you say, stateless, then it
sounds like a "policer" is a separate object that's being created at the
same time as the filter. Is there any other way to create a "policer"
object, or do they only exist when attached to filters? When working
with egress traffic control, can you attach policers to filters, and how
would they interact with the classes?

> > BTW, we are working with a stock RedHat 7.3 2.4.20-18.7 kernel, and we
> > are VERY reluctant to apply patches of any kind, so we're just going to
> > work with whatever is available in the bits we download. We're
> > considering moving up to 2.4.21 to get IMQ, but that doesn't appear to
> > be available for RedHat 7.3 yet - in fact, this may force us to RedHat
> > 9.0 (not a bad thing, really).
> Why don't you download the kernel source 7.3 2.4.20-18.7 from RH and the 
> config file?  So you have the same source as the kernel you trust.  If you 
> trust that source, you can pacth it with what ever you want.

Our problem is not really with "trust". Our problem is that we have very
limited staff, and we don't want to open up a whole new area of effort.
Using "stock" kernels and not having to compile them means that none of
us have to go through the effort of managing all that. We have the
expertise, we just don't have the time.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Stef Coene]

On Wednesday 06 August 2003 20:41, Patrick Turley wrote:
> > > The fact that the filters are metering traffic flows implies that they
> > > have are stateful. When using filters with egress queue hierarchies, it
> > > was my understanding that no state was needed since all they do is
> > > direct packets into classes. It sounds like my current understanding is
> > > quite wrong.
> >
> > I don't think the filters are statefull.  If you use policers, you only
> > rate limit the filters.  The filters itself have no idea about the
> > traffic flows.
>
> Then there are two further questions (the first of which I actually
> should have asked before):
>
> 1) When working with egress traffic control, filters are attached to
> classes. The documents I've read so far tell me that the ingress qdisc
> is classless so, on the face of it, filters shouldn't be useable at all.
> Please tell me which part of this I have badly misunderstood.
Ingress is indeed classless, but you can filters on it.  It's not important 
where the filters point to because there are no classes.  But you can add 
policers to the filters.

> 2) Since the filters themselves are, as you say, stateless, then it
> sounds like a "policer" is a separate object that's being created at the
> same time as the filter. Is there any other way to create a "policer"
> object, or do they only exist when attached to filters? When working
> with egress traffic control, can you attach policers to filters, and how
> would they interact with the classes?
You have to see the policer as a small tbf qdisc that exists on his own.  If 
you add the policer to a filter, all packets are "queued" in the policer and 
throttled.
You can also use the same policer on different filters.  So the packets from 
different filters are "queued" in the same policer.  To do so, you need the 
tcindex option.

I will dig in my mail archive and put online on www.docum.org what I can find.

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Stef Coene]

On Wednesday 06 August 2003 21:23, Stef Coene wrote:
> I will dig in my mail archive and put online on www.docum.org what I can
> find.
The only thing I found was :
http://www.docum.org/stef.coene/qos/faq/cache/69.html

Stef

-- 

stef.coene@docum.org
 "Using Linux as bandwidth manager"
     http://www.docum.org/
     #lartc @ irc.oftc.net

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Martin A. Brown]

Hello all,

I played a bit with the ingress qdisc after seeing Patrick and Stef
talking about it and came up with a few notes and a few questions.

 : The ingress qdisc itself has no parameters.  The only thing you can do
 : is using the policers.  I have a link with a patch to extend this :
 : http://www.cyberus.ca/~hadi/patches/action/ Maybe this can help.
 :
 : I have some more info about ingress in my mail files, but I have to
 : sort it out and put it somewhere on docum.org.  But I still didn't
 : found the the time to do so.

Regarding policers and the ingress qdisc.  I have never used them before
today, but have the following understanding.

About the ingress qdisc:

  - ingress qdisc (known as "ffff:") can't have any children classes
    (hence the existence of IMQ)
  - the only thing you can do with the ingress qdisc is attach filters

About filtering on the ingress qdisc:

  - since there are no classes to which to direct the packets, the only
    reasonable option (reasonable, indeed!) is to drop the packets
  - with clever use of filtering, you can limit particular traffic
    signatures to particular uses of your bandwidth

Here's an example of using an ingress policer to limit inbound traffic
from a particular set of IPs on a per IP basis.  In this case, traffic
from each of these source IPs is limited to a T1's worth of bandwidth.
Note that this means that this host can receive up to 1536kbit (768kbit +
768kbit) worth of bandwidth from these two source IPs alone.

# -- start of script
#! /bin/ash
#
# -- simulate a much smaller amount of bandwidth than the 100MBit interface
#
RATE\x1536kbit
DEV=eth0
SOURCES="10.168.53.2/32 10.168.73.10/32 10.168.28.20/32"

# -- attach our ingress qdisc
#
tc qdisc add dev $DEV ingress

# -- cap bandwidth from particular source IPs
#

for SOURCE in $SOURCES ; do

  tc filter add dev $DEV parent ffff: protocol ip   \
    u32 match ip src $SOURCE flowid :1              \
    police rate $RATE mtu 12k burst 10k drop

done

# -- end of script

Now, if you are using multiple public IPs on your masquerading/SNAT host,
you can use "u32 match ip dst $PER_IP" with a drop action to force a
particular rate on inbound traffic to that IP.

My entirely unquantified impression is that latency suffers as a result,
but traffic is indeed bandwidth limited.

Just a few notes of dissection:

  tc filter add dev $DEV   # -- the usual beginnings
    parent ffff:           # -- the ingress qdisc itself
    protocol ip            # -- more preamble  | make sure to visit
    u32 match ip           # -- u32 classifier | http://lartc.org/howto/
    src $SOURCE            # -- could also be "dst $SOME_LOCAL_IP"
    flowid :1              # -- ??? (but it doesn't work without this)
    police rate $RATE      # -- put a policer here
    mtu 12k burst 10k      # -- ???
    drop                   # -- drop packets exceeding our police params

Maybe a guru or two out there (Stef?, Bert?, Jamal?, Werner?) can explain
why mtu needs to be larger than 1k (didn't work for me anyway) and also
how these other parameters should be used.

I'll add a few questions:

  Why does policing fail entirely without "flowid :1"?  (There's no flow!)
  Why does the policer drop all traffic if mtu is small?
  How are mtu and burst used?

Oh yes, and "Ah, that's linux!"

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Martin A. Brown]

Patrick,

As far as I'm concerned, these are two very good questions.

 : 1) When working with egress traffic control, filters are attached to
 :    classes. The documents I've read so far tell me that the ingress
 :    qdisc is classless so, on the face of it, filters shouldn't be
 :    useable at all.  Please tell me which part of this I have badly
 :    misunderstood.

Agreed....it doesn't make sense, to me, except as code reuse.  Any takers
care to explain why this is?

 : 2) Since the filters themselves are, as you say, stateless, then it
 :    sounds like a "policer" is a separate object that's being created at
 :    the same time as the filter. Is there any other way to create a
 :    "policer" object, or do they only exist when attached to filters?

My understand is that the policers only exist attached to filters.  They
are separate, but not able to be generated without a filter.  I think
Stef's description isn't so bad....a policer is similar to a token bucket
filter (TBF) which is attached to a filter.  I don't know what it looks
like inside the policing code, so I'll have to assume that this is
correct.  From the outside of the black box, they are pretty much rate
limiters which you can attach to any "tc filter" command.

 :    When working with egress traffic control, can you attach policers
 :    to filters,

Yes.

 :     and how would they interact with the classes?

When a new packet arrives on an output queue prior to outbound traffic
control (but after netfilter POSTROUTING), the packet will first traverse
the filters attached to qdisc 1:0 to learn of its destination flowid.
Here, you could have a number of different filters which classify and/or
reclassify the packet into different output classes/qdiscs.  You could
optionally drop packets with a filter here, just as you could drop packets
with filters on the ingress qdisc.

These policers will be VERB (executed? obeyed?) before the packet ever
enters the classes and sub-classes.

-Martin

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Patrick Turley]

First of all, thank you Martin - this is fabulously helpful.

On Wed, 2003-08-06 at 20:18, Martin A. Brown wrote:
> Hello all,
> 
> I played a bit with the ingress qdisc after seeing Patrick and Stef
> talking about it and came up with a few notes and a few questions.
> 
> ...
> 
> About filtering on the ingress qdisc:
> 
>   - since there are no classes to which to direct the packets, the only
>     reasonable option (reasonable, indeed!) is to drop the packets
> 
>   - with clever use of filtering, you can limit particular traffic
>     signatures to particular uses of your bandwidth

That sounds like a correct statement about the filters themselves. But
it appears that we don't have to just drop the packets, because the
policers we attach to the filters CAN queue packets. The following quote
is from one of Stef's earlier replies:

> You have to see the policer as a small tbf qdisc that exists on his own.  If 
> you add the policer to a filter, all packets are "queued" in the policer and 
> throttled.

Martin continues...

> Here's an example of using an ingress policer to limit inbound traffic
> from a particular set of IPs on a per IP basis.  In this case, traffic
> from each of these source IPs is limited to a T1's worth of bandwidth.
> Note that this means that this host can receive up to 1536kbit (768kbit +
> 768kbit) worth of bandwidth from these two source IPs alone.
> 
> # -- start of script
> #! /bin/ash
> #
> # -- simulate a much smaller amount of bandwidth than the 100MBit interface
> #
> RATE\x1536kbit
> DEV=eth0
> SOURCES="10.168.53.2/32 10.168.73.10/32 10.168.28.20/32"
> 
> # -- attach our ingress qdisc
> #
> tc qdisc add dev $DEV ingress

I don't see a handle argument here. I presume that, since the ingress
handle for an interface MUST be ffff, that the handle argument is
optional.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Patrick Turley]

On Wed, 2003-08-06 at 20:37, Martin A. Brown wrote:
>  : 2) Since the filters themselves are, as you say, stateless, then it
>  :    sounds like a "policer" is a separate object that's being created at
>  :    the same time as the filter. Is there any other way to create a
>  :    "policer" object, or do they only exist when attached to filters?
> 
> My understand is that the policers only exist attached to filters.  They
> are separate, but not able to be generated without a filter.  I think
> Stef's description isn't so bad....a policer is similar to a token bucket
> filter (TBF) which is attached to a filter.  I don't know what it looks
> like inside the policing code, so I'll have to assume that this is
> correct.  From the outside of the black box, they are pretty much rate
> limiters which you can attach to any "tc filter" command.

From what Stef says, the documentation on the TC_INDEX qdisc illuminates
the use and structure of policers. I've looked TC_INDEX only very
briefly. I mention this here only so that anyone else who is interested
will have the pointer.

>  :    When working with egress traffic control, can you attach policers
>  :    to filters,
> 
> Yes.
> 
>  :     and how would they interact with the classes?
> 
> When a new packet arrives on an output queue prior to outbound traffic
> control (but after netfilter POSTROUTING), the packet will first traverse
> the filters attached to qdisc 1:0 to learn of its destination flowid.
> Here, you could have a number of different filters which classify and/or
> reclassify the packet into different output classes/qdiscs.  You could
> optionally drop packets with a filter here, just as you could drop packets
> with filters on the ingress qdisc.
> 
> These policers will be VERB (executed? obeyed?) before the packet ever
> enters the classes and sub-classes.

OK - This is very interesting to me. I understand the mechanism whereby
filters guide packets to the classes for which they are destined. That
all makes sense.

Based on what I've read here, it sounds like you can interpose a policer
between the filter itself and the class to which it is directing
packets. Given that classes already have access to the full range of
qdiscs, with all their marvelous dials and knobs, interposing a policer
seems superfluous. Have I misunderstood?


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Parameters for the ingress qdisc?

[Martin A. Brown]

 : OK - This is very interesting to me. I understand the mechanism whereby
 : filters guide packets to the classes for which they are destined. That
 : all makes sense.
 :
 : Based on what I've read here, it sounds like you can interpose a
 : policer between the filter itself and the class to which it is
 : directing packets. Given that classes already have access to the full
 : range of qdiscs, with all their marvelous dials and knobs, interposing
 : a policer seems superfluous. Have I misunderstood?

Well, I wouldn't say superfluous.....and I don't think you have
misunderstood, but perhaps you don't yet see the rationale for a policer.
Let me suggest a scenario.  You have users who send mail.*  You wish
to guarantee a certain small percentage of your 2Mbit to only the SMTP
traffic, so that at least 256kbit is reserved for mail at any time.
Additionally, you want to put any SMTP flows above this 256kbit into a
general consumption class.

Well, your policer reclassifies any traffic above a given rate into the
general consumption class.

That's just one way of using a policer.  My understanding is that policers
are more commonly used for DiffServ domains and limiting inbound traffic,
although there's nothing preventing them from being used in ingress
traffic situations.

-Martin

 * I know it sounds incredible for me to suggest that a network admin
   might have users, but suspend your disbelief for now, if you can.

-- 
Martin A. Brown --- SecurePipe, Inc. --- mabrown@securepipe.com

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/