Good qdisc for routers?

Good qdisc for routers?

[Dennis Jacobfeuerborn]

Hi,
I'm setting up a linux box as a router/firewall and I'm wondering if 
there are any best practices with regards to network settings that 
should be tweaked in this situation.
In particular I'm wondering if the default pfifo_fast is susceptible to 
a single flow flooding the buffer and causing other flows to lose 
packets? Is there an alternative that would work better in terms of 
fairness but doesn't create too much overhead or is overly complicated 
to configure?
The system has 1Gbit interfaces.

Regards,
   Dennis

Re: Good qdisc for routers?

[Dennis Jacobfeuerborn]

On 05.11.2013 19:06, Remy Mudingay wrote:
> Hi Dennis,
>
> I  highly recommend trying out codel or fq_codel. I've been using
> fq_codel for well over a year now. I have just over a thousand users
> going through the router. I use it mainly because I wanted in-queue
> packet delays statistics but I didn't switch from pfifo_fast because of
> any perceived problems.
> Anyway, You won't need to patch your current Linux install as long as
> it's version 3.3 or higher.
>
> I use the following two lines for each interface.
>
> tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn
> tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash
> keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor
> 10242 perturb 300 baseclass 1:1
>
> You can get more in depth information from here
> http://www.bufferbloat.net/projects/codel/wiki.

Codel seems more geared towards low latency. The following statement 
from the wiki doesn't make it sound like this is the right choice here:

"People have tried to run CoDel in very big routers, with hundreds of 
simultaneous flows, a situation not simulated in advance. There, it 
isn't controlling the queue the way it should: whether this is a problem 
with the algorithm, or the implementation, is not yet understood.

It is clear that unmodified, CoDel is not appropriate for AQM of traffic 
inside a data center; it does not react in a timely enough fashion. 
Whether the modifications of the ideas in CoDel will solve this problem 
is not yet known. Again, this is an area which CoDel was not designed to 
solve or it simulated in before publication."

Regards,
   Dennis

Re: Good qdisc for routers?

[Remy Mudingay]

Hi Dennis,

I read the same article but since it mentioned datacenter I understood this to not apply to Internet bound traffic. However even on a gigabit WAN link codel (in my case fq_codel) could improve the responsiveness of competing flows. A few open source distributions (openwrt,dd-wrt and gargoyle).

By the way, why do you want to move away from the default qdisc?

Cheers,

Remy.

Sent from my iPhone 

> On Nov 5, 2013, at 19:42, Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
> 
>> On 05.11.2013 19:06, Remy Mudingay wrote:
>> Hi Dennis,
>> 
>> I  highly recommend trying out codel or fq_codel. I've been using
>> fq_codel for well over a year now. I have just over a thousand users
>> going through the router. I use it mainly because I wanted in-queue
>> packet delays statistics but I didn't switch from pfifo_fast because of
>> any perceived problems.
>> Anyway, You won't need to patch your current Linux install as long as
>> it's version 3.3 or higher.
>> 
>> I use the following two lines for each interface.
>> 
>> tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn
>> tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash
>> keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor
>> 10242 perturb 300 baseclass 1:1
>> 
>> You can get more in depth information from here
>> http://www.bufferbloat.net/projects/codel/wiki.
> 
> Codel seems more geared towards low latency. The following statement from the wiki doesn't make it sound like this is the right choice here:
> 
> "People have tried to run CoDel in very big routers, with hundreds of simultaneous flows, a situation not simulated in advance. There, it isn't controlling the queue the way it should: whether this is a problem with the algorithm, or the implementation, is not yet understood.
> 
> It is clear that unmodified, CoDel is not appropriate for AQM of traffic inside a data center; it does not react in a timely enough fashion. Whether the modifications of the ideas in CoDel will solve this problem is not yet known. Again, this is an area which CoDel was not designed to solve or it simulated in before publication."
> 
> Regards,
>  Dennis

Re: Good qdisc for routers?

[Dennis Jacobfeuerborn]

Hi Remy,
from what I understand the default pfifo_fast qdisc isn't particulary 
fair when a single flow or a few flows flood the interface/buffer with 
packets so I am wondering if qdiscs like QFQ, SFQ or CHOKe could be 
improvements compared to the default "dumb" pfifo one.

Regards,
   Dennis

On 05.11.2013 20:33, Remy Mudingay wrote:
> Hi Dennis,
>
> I read the same article but since it mentioned datacenter I understood this to not apply to Internet bound traffic. However even on a gigabit WAN link codel (in my case fq_codel) could improve the responsiveness of competing flows. A few open source distributions (openwrt,dd-wrt and gargoyle).
>
> By the way, why do you want to move away from the default qdisc?
>
> Cheers,
>
> Remy.
>
> Sent from my iPhone
>
>> On Nov 5, 2013, at 19:42, Dennis Jacobfeuerborn <dennisml@conversis.de> wrote:
>>
>>> On 05.11.2013 19:06, Remy Mudingay wrote:
>>> Hi Dennis,
>>>
>>> I  highly recommend trying out codel or fq_codel. I've been using
>>> fq_codel for well over a year now. I have just over a thousand users
>>> going through the router. I use it mainly because I wanted in-queue
>>> packet delays statistics but I didn't switch from pfifo_fast because of
>>> any perceived problems.
>>> Anyway, You won't need to patch your current Linux install as long as
>>> it's version 3.3 or higher.
>>>
>>> I use the following two lines for each interface.
>>>
>>> tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn
>>> tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash
>>> keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor
>>> 10242 perturb 300 baseclass 1:1
>>>
>>> You can get more in depth information from here
>>> http://www.bufferbloat.net/projects/codel/wiki.
>>
>> Codel seems more geared towards low latency. The following statement from the wiki doesn't make it sound like this is the right choice here:
>>
>> "People have tried to run CoDel in very big routers, with hundreds of simultaneous flows, a situation not simulated in advance. There, it isn't controlling the queue the way it should: whether this is a problem with the algorithm, or the implementation, is not yet understood.
>>
>> It is clear that unmodified, CoDel is not appropriate for AQM of traffic inside a data center; it does not react in a timely enough fashion. Whether the modifications of the ideas in CoDel will solve this problem is not yet known. Again, this is an area which CoDel was not designed to solve or it simulated in before publication."
>>
>> Regards,
>>   Dennis

Re: Good qdisc for routers?

[Nicolas Sebrecht]

The 05/11/13, Dennis Jacobfeuerborn wrote:
> Hi Remy,
> from what I understand the default pfifo_fast qdisc isn't particulary 
> fair when a single flow or a few flows flood the interface/buffer with 
> packets so I am wondering if qdiscs like QFQ, SFQ or CHOKe could be 
> improvements compared to the default "dumb" pfifo one.

You are right the pfifo_fast is not fair at all. 

Now, unless you already know that you will hit some real use cases of
such floods causing damages to the flows I would recommend you to stick
to the default pfifo_fast which work great most of the time. Each qdisc
comes with advantages and inconveniences.

Of course, if you experiment some network issues the correct thing to
do might be to tune some queueing disciplines. Though in practice, this
is not even sure that this would be the root cause as network problems
can come from a lot of other things (offloading, hardware, drivers,
etc).

Most of the time, I think people tune qdiscs because of real issues or a
strong policy and I guess this is the good thing to do. Doing it to
prevent /potential/ issues looks like avoidable pain and administration
overhead, to me.

Have fun,

-- 
Nicolas Sebrecht

Re: Good qdisc for routers?

[Dave Taht]

On 05.11.2013 19:06, Remy Mudingay wrote:

>>Hi Dennis,

>>I  highly recommend trying out codel or fq_codel. I've been using
>>fq_codel for well over a year now. I have just over a thousand users
>>going through the router. I use it mainly because I wanted in-queue
>>packet delays statistics but I didn't switch from pfifo_fast because of
>>any perceived problems.
>>Anyway, You won't need to patch your current Linux install as long as
>>it's version 3.3 or higher.

>>I use the following two lines for each interface.

>>tc qdisc add dev eth0 root handle 1: fq_codel quantum 1514 flows 1024 noecn
>>tc filter add dev eth0 prio 1 protocol all parent 1: handle 1 flow hash
>>keys nfct-src,nfct-dst,nfct-proto,nfct-proto-src,nfct-proto-dst divisor
>>10242 perturb 300 baseclass 1:1

Unless you *really* care about preserving the pre-natted ip/port
information in the hash, the second filter is unneeded. I am not huge
on perturbation, in this case it would mean that all the per flow
codel state is lost ever 300 seconds. fq_codel basically has an
identical hash by default, no perturbation, but it randomizes its
default hash on init time so it's unpredictable that way.

I would certainly like it if there was a tc command to "randomize" a
custom hash like your second line there for fq_codel.

I generally have no problem leaving ecn on presently. It is rarely
used, and even more rarely abused. When used, it helps.

>>You can get more in depth information from here
>>http://www.bufferbloat.net/projects/codel/wiki.

>Codel seems more geared towards low latency. The following statement from the >wiki doesn't make it sound like this is the right choice here:

We are pretty happy with fq_codel in the range 4Mbit to 10Gbit. Why
not go for low latency if you can get 100% utilization?

>"People have tried to run CoDel in very big routers, with hundreds of >simultaneous flows, a situation not simulated in advance. There, it isn't >controlling the queue the way it should: whether this is a problem with the >algorithm, or the implementation, is not yet understood."

Several ideas are out of context here. I will ask kathie to clarify.

1) The Codel ns2 model has some problems at hundreds of simultaneous
full rate flows.
2) fq_codel, which splits up flows into up to 64k queues each managed
by codel, does not.
3) "Use fq_codel, not codel. It's an across the board win." - Van Jacobson

http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos

>It is clear that unmodified, CoDel is not appropriate for AQM of traffic inside a >data center; it does not react in a timely enough fashion. Whether the >modifications of the ideas in CoDel will solve this problem is not yet known. >Again, this is an area which CoDel was not designed to solve or it simulated in >before publication."

And here what kathie meant was that codel was designed for edge
networks to "do no harm" to be on by default everywhere, at RTTs
ranging from a few ms to hundreds. Within-the-data center speeds are
well below that, and the reaction times in *codel* are too slow.

Since the paper, ecn support was added to codel and fq_codel and works
well, and by tweaking two variables (target and interval) down to
values that make
more sense inside the data center, the aqm stuff works increasingly better,
and the "fq" portion of fq_codel helps in all cases at all rates, and
improves codel's effective reaction time as well. Yes there is more
work to be done inside the data center. We've spent more time trying
to improve rates below 4mbit than above 10gig (see nfq_codel)

However:

At this point I'm pretty happy with the new "fq" scheduler for linux
hosts in linux 3.12, and fq_codel on linux based hosts and routers, at
nearly any speed. Try it, exercise it, you'll like it.

Even at gig+ speeds you get back a more than a few ms of latency.

http://www.bufferbloat.net/projects/codel/wiki/Best_practices_for_benchmarking_Codel_and_FQ_Codel

http://www.bufferbloat.net/projects/codel/wiki/RRUL_Rogues_Gallery
-- 
Dave Täht

Fixing bufferbloat with cerowrt: http://www.teklibre.com/cerowrt/subscribe.html