[LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[Greg Stark]

Damion de Soto <damion@snapgear.com> writes:

> You can create different ingress policers that only match specific ports, and
> give them different priorities, but that still won't work as well as using IMQ,
> or if your box is a gateway (and you are only shaping traffic going through it),
> then you can use egress queues on the LAN interface.

For some reason that hadn't occurred to me. That should work just fine. I
guess I should mark the packets in iptables to avoid throttling traffic from
gateway itself, or does match see the external ip?

IMQ does seem like a handy tool, but why is there a distinction at all between
egress and ingress qdiscs at all? Why not just allow people to attach HTB as
an ingress qdisc directly?


I suppose in an ideal world the best thing would be to receive the packets,
hand them to user-space, but not mark them as received, ie, not ack them. That
avoids introducing any loss from the user point of view but still slows the
flow down. But that seems fiddly and would only work for TCP I guess.

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

[LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[Damion de Soto]

Greg,
> For some reason that hadn't occurred to me. That should work just fine. I
> guess I should mark the packets in iptables to avoid throttling traffic from
> gateway itself, or does match see the external ip?
The only (common) time you need to use iptables to mark traffic, is when you're using 
  egress qdiscs on your outbound (WAN) interface (to shape upload speeds) and want to 
filter based on private (LAN) IP address.

> IMQ does seem like a handy tool, but why is there a distinction at all between
> egress and ingress qdiscs at all? Why not just allow people to attach HTB as
> an ingress qdisc directly?
because you can't shape inbound traffic.  Shaping works by delaying the transmission, and
you can't delay packets that haven't arrived yet. Ingress policing just drops 
packets, and hopes the sender will slow down.

> I suppose in an ideal world the best thing would be to receive the packets,
> hand them to user-space, but not mark them as received, ie, not ack them. That
> avoids introducing any loss from the user point of view but still slows the
> flow down. But that seems fiddly and would only work for TCP I guess.
That's an interesting idea, but yeah... i think it might be a bit hard to implement, 
and violate far too many RFCs...

regards,

-- 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Damion de Soto - Software Engineer  email:     damion@snapgear.com
SnapGear - A CyberGuard Company ---    ph:         +61 7 3435 2809
  | Custom Embedded Solutions          fax:         +61 7 3891 3630
  | and Security Appliances            web: http://www.snapgear.com
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ---  Free Embedded Linux Distro at   http://www.snapgear.org  ---
_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

[LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[Greg Stark]

Damion de Soto <damion@snapgear.com> writes:

> Greg,
> > For some reason that hadn't occurred to me. That should work just fine. I
> > guess I should mark the packets in iptables to avoid throttling traffic from
> > gateway itself, or does match see the external ip?
>
> The only (common) time you need to use iptables to mark traffic, is when
> you're using egress qdiscs on your outbound (WAN) interface (to shape upload
> speeds) and want to filter based on private (LAN) IP address.

Well I'll be doing the opposite here, putting an egress qdisc on my LAN
interface limiting the downstream bandwidth. So I would have the equivalent
need, to distinguish traffic originating from outside vs traffic originating
from the router so I can still scp kernels over to the router without being
limited. Hm, though in practice I hardly ever scp things _from_ the router so
perhaps I don't care.

> > IMQ does seem like a handy tool, but why is there a distinction at all between
> > egress and ingress qdiscs at all? Why not just allow people to attach HTB as
> > an ingress qdisc directly?
>
> because you can't shape inbound traffic. Shaping works by delaying the
> transmission, and you can't delay packets that haven't arrived yet. Ingress
> policing just drops packets, and hopes the sender will slow down.

Well ultimately all shaping works by dropping packets. Merely delaying
transmission isn't going to slow down anything in the long run, just increase
the pipeline. You can delay and/or drop them after they've arrived just as
easily. Though it would have to be before they're ack'd and delivered to the
user. That's basically what IMQ does, I'm just saying perhaps that should just
work instead of requiring a fake interface.

Hm, I wonder if I want RED or something similar to ensure packets get dropped
fast enough instead of filling HTB queues and then dropping.

> > I suppose in an ideal world the best thing would be to receive the packets,
> > hand them to user-space, but not mark them as received, ie, not ack them. That
> > avoids introducing any loss from the user point of view but still slows the
> > flow down. But that seems fiddly and would only work for TCP I guess.
>
> That's an interesting idea, but yeah... i think it might be a bit hard to
> implement, and violate far too many RFCs...

My original thought was to just drop an ACK packet, which would clearly not
violate any assumptions. But if there's data in the ACK packet that could have
negative effects. I suspect this lies well into Bad Idea(tm) realm, I'm just
thinking out loud.

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[Jason Boxman]

On Tuesday 08 June 2004 23:33, Greg Stark wrote:
> Damion de Soto <damion@snapgear.com> writes:
<snip>
> >
> > because you can't shape inbound traffic. Shaping works by delaying the
> > transmission, and you can't delay packets that haven't arrived yet.
> > Ingress policing just drops packets, and hopes the sender will slow down.
>
> Well ultimately all shaping works by dropping packets. Merely delaying
> transmission isn't going to slow down anything in the long run, just
> increase the pipeline. You can delay and/or drop them after they've arrived
> just as easily. Though it would have to be before they're ack'd and
> delivered to the user. That's basically what IMQ does, I'm just saying
> perhaps that should just work instead of requiring a fake interface.

Ultimately, packets from a misbehaving flow can be dropped, but it does not 
always come to a drop.  When you shape on egress, you force applications on 
the local network to throttle back, believing they're sending as fast as the 
receiver can receive.  As you delay, TCP figures it out.  Contrast that with 
ingress, where the packets you want to delay are already on their way.

> Hm, I wonder if I want RED or something similar to ensure packets get
> dropped fast enough instead of filling HTB queues and then dropping.

If you're curious about RED, here's a possible example implementation for 
ingress policing:

http://digriz.org.uk/jdg-qos-script/

<snip>

-- 

Jason Boxman
Perl Programmer / *NIX Systems Administrator
Shimberg Center for Affordable Housing | University of Florida
http://edseek.com/ - Linux and FOSS stuff

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

[LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[Greg Stark]

Jason Boxman <jasonb@edseek.com> writes:

> On Tuesday 08 June 2004 23:33, Greg Stark wrote:
> >
> > Well ultimately all shaping works by dropping packets. Merely delaying
> > transmission isn't going to slow down anything in the long run, just
> > increase the pipeline. You can delay and/or drop them after they've arrived
> > just as easily. Though it would have to be before they're ack'd and
> > delivered to the user. That's basically what IMQ does, I'm just saying
> > perhaps that should just work instead of requiring a fake interface.
> 
> Ultimately, packets from a misbehaving flow can be dropped, but it does not 
> always come to a drop.  When you shape on egress, you force applications on 
> the local network to throttle back, believing they're sending as fast as the 
> receiver can receive.  As you delay, TCP figures it out.  

I don't think so. It may look like that's what's happening, but at least for
TCP I don't think it works that way. As long as packets aren't being dropped
then TCP will just continue to grow the window, interpreting this delay as
just a longer pipeline that needs filling. Applications will be slowed down
temporarily because it takes time to do this, but they'll eventually be
outputting data just as fast as an application without a shaper.

Only when packets get dropped or are delayed so long that the client
retransmits does TCP scale back the transmit window. And only when that
happens does the client see any reduced bandwidth. So if your shaper isn't
dropping packets it's just evening out the flow of data, not actually
affecting the net rate the clients can pump out data.

> Contrast that with ingress, where the packets you want to delay are already
> on their way.

Well on egress the packets are "already on their way" as well, after all.
They're just haven't gone as many hops. Even for locally generated traffic the
egress qdisc is being run after the data packetized and ready to go.

> > Hm, I wonder if I want RED or something similar to ensure packets get
> > dropped fast enough instead of filling HTB queues and then dropping.
> 
> If you're curious about RED, here's a possible example implementation for 
> ingress policing:
> 
> http://digriz.org.uk/jdg-qos-script/

This is an interesting script. It looks like a successor to wondershaper. But
I'm a bit too deep in my own re-implementation of wondershaper now to start
over.

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: how flexible is ingress traffic policing to

[Sanjay Arora]

On Wed, 2004-06-09 at 09:03, Greg Stark wrote:
> Damion de Soto <damion@snapgear.com> writes:
> 

> > because you can't shape inbound traffic. Shaping works by delaying the
> > transmission, and you can't delay packets that haven't arrived yet. Ingress
> > policing just drops packets, and hopes the sender will slow down.
>
Sorry to interrupt the flow, especially being a newbie, but won´t the
sender just retransmit the dropped packets at the same rate? I am not so
thorogh with TCP/IP, but is there something in the protocol that speeds
or slows the transmission.

Please do explain in the TCP/IP for complete idiots terminology ;-))

With best regards.
Sanjay.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

[LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[Greg Stark]

Sanjay Arora <skpobox@gawab.com> writes:

> Sorry to interrupt the flow, especially being a newbie, but won´t the
> sender just retransmit the dropped packets at the same rate? 

no.

> I am not so thorogh with TCP/IP, but is there something in the protocol that
> speeds or slows the transmission.

yes.

> Please do explain in the TCP/IP for complete idiots terminology ;-))

You could do worse than this:

http://www.thinkingsecure.com/docs/TCPIP-Illustrated-1/tcp_time.htm#21_0

-- 
greg

_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: how flexible is ingress traffic policing to bandwidth limit?

[Jason Boxman]

On Wednesday 09 June 2004 16:09, Greg Stark wrote:
> Sanjay Arora <skpobox@gawab.com> writes:
> > Sorry to interrupt the flow, especially being a newbie, but won´t the
> > sender just retransmit the dropped packets at the same rate?
>
> no.
>
> > I am not so thorogh with TCP/IP, but is there something in the protocol
> > that speeds or slows the transmission.
>
> yes.
>
> > Please do explain in the TCP/IP for complete idiots terminology ;-))
>
> You could do worse than this:
>
> http://www.thinkingsecure.com/docs/TCPIP-Illustrated-1/tcp_time.htm#21_0

I have to admit, I am thus far thoroughly enjoying:

Jacobson, V., Congestion Avoidance and Control. Proceedings of ACM SIGCOMM 
'88. August 1988, p. 314-329.

It's also available in Postscript format:

ftp://ftp.ee.lbl.gov/papers/congavoid.ps.Z

For those who don't know how TCP handles congestion, this is a rather 
excellent read.


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/

Re: [LARTC] Re: how flexible is ingress traffic policing to bandwidth

[Andy Furniss]

Greg Stark wrote:
> Jason Boxman <jasonb@edseek.com> writes:
> 
> 
>>On Tuesday 08 June 2004 23:33, Greg Stark wrote:
>>
>>>Well ultimately all shaping works by dropping packets. Merely delaying
>>>transmission isn't going to slow down anything in the long run, just
>>>increase the pipeline. You can delay and/or drop them after they've arrived
>>>just as easily. Though it would have to be before they're ack'd and
>>>delivered to the user. That's basically what IMQ does, I'm just saying
>>>perhaps that should just work instead of requiring a fake interface.
>>
>>Ultimately, packets from a misbehaving flow can be dropped, but it does not 
>>always come to a drop.  When you shape on egress, you force applications on 
>>the local network to throttle back, believing they're sending as fast as the 
>>receiver can receive.  As you delay, TCP figures it out.  
> 
> 
> I don't think so. It may look like that's what's happening, but at least for
> TCP I don't think it works that way. As long as packets aren't being dropped
> then TCP will just continue to grow the window, interpreting this delay as
> just a longer pipeline that needs filling. Applications will be slowed down
> temporarily because it takes time to do this, but they'll eventually be
> outputting data just as fast as an application without a shaper.

Eventually in the egress case either you will run out of buffer and 
drop, or your queue is big enough for the advertised window (say 
32k-64k) and tcp will only add a packet for everyone acked - you decide 
when the packets go so you have good control. If you drop, the sender 
reduces it's cwin and only slowly tries to increase until another drop.

> 
> Only when packets get dropped or are delayed so long that the client
> retransmits does TCP scale back the transmit window. And only when that
> happens does the client see any reduced bandwidth. So if your shaper isn't
> dropping packets it's just evening out the flow of data, not actually
> affecting the net rate the clients can pump out data.

Once you have dropped the rate you release packets does controll the 
flow and if it's slow the cwin will only grow slowly as it's clocked by 
acks (I think).

> 
> 
>>Contrast that with ingress, where the packets you want to delay are already
>>on their way.
> 
> 
> Well on egress the packets are "already on their way" as well, after all.
> They're just haven't gone as many hops. Even for locally generated traffic the
> egress qdisc is being run after the data packetized and ready to go.
> 

I think what Jason was getting at is that on egress you have total 
control - it doesn't matter what TCP does.

Ingress shaping is harder as packets are already headed for you - in the 
case of P2P possibly quite alot - it's easier to shape when peering with 
"real" servers. Though in both cases, the fact that TCP slow (but sort 
of exponential) start overshoots bandwidth hurts ingress shaping, but 
not egress.

Andy.

> 
>>>Hm, I wonder if I want RED or something similar to ensure packets get
>>>dropped fast enough instead of filling HTB queues and then dropping.
>>
>>If you're curious about RED, here's a possible example implementation for 
>>ingress policing:
>>
>>http://digriz.org.uk/jdg-qos-script/
> 
> 
> This is an interesting script. It looks like a successor to wondershaper. But
> I'm a bit too deep in my own re-implementation of wondershaper now to start
> over.
> 


_______________________________________________
LARTC mailing list / LARTC@mailman.ds9a.nl
http://mailman.ds9a.nl/mailman/listinfo/lartc HOWTO: http://lartc.org/