Re: Linux the best hardware for routing with 10 and 40G network adapter

Re: Linux the best hardware for routing with 10 and 40G network adapter

[Dave Taht]

On Thu, Mar 19, 2015 at 10:21 AM, Robert LeBlanc <robert@leblanc.us> wrote:
>
> I'm not familiar with fq_codel, but it doesn't look to have priority
> queues which is one of our primary concerns.

You generally don't need priority queues per se' with fq_codel. See
more below. You CAN, if you wish, use specialized tc rules for priority
traffic.

We have generally found that by using fair queueing (flow queuing), and
an AQM to keep queue lengths short, that nearly all of the perceived need
for explicit prioritization vanishes.

try it. you are a single sysctl away from trying it.

> 40 Gb is pretty fast,

Well, an ongoing problem much discussed at the netconf01 conference,
is the tx path is now capable of 40Gbit but the rx path is not.

>
> but
> we are concerned about run away processes in our cluster that could
> saturate uplinks or even the PCI bus at these speeds.

Well, fq_codel is best on your saturated uplinks, to be sure, and it
is very helpful against simple malbehaved loads. For example, we had a
bug in dhcpdv6 that happened after 51 days, totally saturating the
cable uplink with requests. The fq_codel users - never noticed it! The
link was mildly slower, that's all.

http://www.bufferbloat.net/issues/441

I am sure I still have older users of cerowrt out there flooding
comcast with dhcpv6 requests from that particular bug, and not
noticing it.

> We need to keep
> heartbeats and management traffic alive to be able to solve these
> problems.

Heartbeats and management traffic are naturally sparse, and thus
automagically prioritized by fq_codel.

It also uses ECN if you choose, which allows for low latency without
packet loss on hosts that use ecn by default. The relevant sysctls
are:

net.ipv4.tcp_ecn=1
net.core.default_qdisc=fq_codel

Which I usually put in a file in /etc/sysctl.d

and there is also now support for enabling ecn on a per route basis in
the latest kernels, as people are still dubious about being able to
use ecn everywhere. But ecn is not required to be fully enabled for
fq_codel to work it's magic.

But: I know of at least one large web service provider enabling ecn
AND fq_codel by default, everywhere now ( archive.org ) and there are
no doubt countless others.

In your case, with the 10+GigE hardware queues running mq, you can
have a top level tc filter to redirect stuff based on markings into
the "right" hardware queues, and have fq_codel or sch_fq attached
underneath it for each hardware queue to further manage the  traffic.
The latter part of that happens automagically with the first sysctl
above, the tc part is kind of expensive to write rules for, I wish we
had an explicit "diffserv" filter to use instead of one tc rule per
diffserv codepoint you cared about.

sch_fq, is a host only solution, mostly targetted at tcp traffic, and
that *does* support explicit prioritization. It's pretty amazing what
it does...

All I have ever argued on these fronts is merely that people try these
new qdiscs in *their* realistic scenarios and with tools like
netperf-wrapper, to see if they help or not, and (hopefully) share
their results so we can further improve matters.

We are well aware of how much sqm-scripts + fq_codel help cable and
DSL modems in particular (see for example:
http://burntchrome.blogspot.com/2014/05/fixing-bufferbloat-on-comcasts-blast.html
or http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html ),

and we are working on some derivative ideas to improve wifi as part of
the make-wifi-fast effort. (
http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf
)

In other news...

Work is resuming on "sch_cake", which combines htb + fq_codel +
diffserv (and priority) classification into one qdisc (replacing the
sqm-scripts entirely) and I do hope we can get some more testers of
the patches over the next few months. The discussion of "cake" is over
on the codel list on bufferbloat.net where the current patch set is
linked to and the flaws discussed. I do hope this qdisc will also
scale up to 40GigE and be of use in doing traffic engineering at these
speeds, and PLEASE join us in discussing the needed feature set(s) for
your use cases.

And I do generally hope that these algorithms will start appearing the
basic ethernet and switch hardware at some point soon. I am backing a
kickstarter project to get it out there in an FPGA at least... (see:
https://www.kickstarter.com/projects/onetswitch/onetswitch-open-source-hardware-for-networking
)

> It seems that fq_codel really helps with latency and
> bufferbloat (which would also be good for us) and it is possible that
> it would be adequate since we are DSCP marking packets and the
> switches are applying L2 COS. I'll have to look over it. If you have
> some information that would be helpful, I'd be happy to have it.


Oy. More talks and documentation than you can shake a stick at.

we have put a collection of the best up here:
http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos

I enjoy Stephen Hemminger's presos a lot, although my modena talks go
into more detail. Van Jacobson's talk is essential, and the shortest
talk I have ever given on these subjects was the uknof talk.

Most of my work has been focused on fixing the edge of the internet,
rather than stuff in the data center, but I am increasingly interested
in what good all this stuff does in these environments, as I do hope
to see these algorithms start showing up soon in Load Balancers,
BRAS's and CMTSes and ISP gear like that.

Certainly sch_fq is rather widely deployed at at least one large web
service provider, in particular. :) I don't have any data as to where
else it is deployed...


>
> On Wed, Mar 18, 2015 at 4:41 PM, Dave Taht <dave.taht@gmail.com> wrote:
> > On Wed, Mar 18, 2015 at 3:03 PM, Robert LeBlanc <robert@leblanc.us> wrote:
> >>
> >> I can't speak for the specific configuration, but we are having
> >> challenges using the mqprio qdisc on the Intel XL710 adapters. We have
> >> engaged Intel, but we are still having trouble. If you want priority
> >> queuing and using the hardware TX queues, you may have problems with
> >> this family of cards.
> >
> >
> > I am very curious about benchmarks of fq_codel (vs mqprio or
> > pfifo_fast) at these speeds on adaptors like this, preferably driven
> > by repeatable tests like the rrul test in netperf-wrapper.
> >
> > In particular, what happens if only one tx queue is enabled with fq_codel?
> >
> >>
> >> On Wed, Mar 18, 2015 at 1:07 PM, Nieścierowicz Adam
> >> <adam.niescierowicz@justnet.pl> wrote:
> >> > Hi,
> >> >
> >> > in the near future we are planning to purchase new equipment for the router.
> >> >
> >> > Our requirements are:
> >> >
> >> > - four or more 1GbE interfaces
> >> >
> >> > - two-four 10GbE interfaces
> >> >
> >> > - two 40GbE interfaces
> >> >
> >> > - multicast routing with throughput input 1-2 Gb/s output 4x1-2Gb/s
> >> >
> >> > Can someone share your experience what equipment you choose to such a
> >> > configuration
> >> >
> >> > ---
> >> > Thanks,
> >> > Adam Nieścierowicz
> >> > --
> >> > To unsubscribe from this list: send the line "unsubscribe lartc" in
> >> > the body of a message to majordomo@vger.kernel.org
> >> > More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >> --
> >> To unsubscribe from this list: send the line "unsubscribe lartc" in
> >> the body of a message to majordomo@vger.kernel.org
> >> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> >
> >
> > --
> > Dave Täht
> > Let's make wifi fast, less jittery and reliable again!
> >
> > https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb




-- 
Dave Täht
Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb

Re: Linux the best hardware for routing with 10 and 40G network adapter

[Nieścierowicz Adam]

Hi Dave,

attached information look very interesting. if you are looking for 
testers I have the opportunity to test in an environment:
- linux router for customers
- 2K active users
- 1xGbE for uplink
- 3xGbE for customer link
- customer use static routing and pppoe


---
Thanks
Adam Nieścierowicz

W dniu 19.03.2015 18:53, Dave Taht napisał(a):

> On Thu, Mar 19, 2015 at 10:21 AM, Robert LeBlanc <robert@leblanc.us> 
> wrote:
> 
>> I'm not familiar with fq_codel, but it doesn't look to have priority
>> queues which is one of our primary concerns.
> 
> You generally don't need priority queues per se' with fq_codel. See
> more below. You CAN, if you wish, use specialized tc rules for priority
> traffic.
> 
> We have generally found that by using fair queueing (flow queuing), and
> an AQM to keep queue lengths short, that nearly all of the perceived 
> need
> for explicit prioritization vanishes.
> 
> try it. you are a single sysctl away from trying it.
> 
>> 40 Gb is pretty fast,
> 
> Well, an ongoing problem much discussed at the netconf01 conference,
> is the tx path is now capable of 40Gbit but the rx path is not.
> 
>> but
>> we are concerned about run away processes in our cluster that could
>> saturate uplinks or even the PCI bus at these speeds.
> 
> Well, fq_codel is best on your saturated uplinks, to be sure, and it
> is very helpful against simple malbehaved loads. For example, we had a
> bug in dhcpdv6 that happened after 51 days, totally saturating the
> cable uplink with requests. The fq_codel users - never noticed it! The
> link was mildly slower, that's all.
> 
> http://www.bufferbloat.net/issues/441 [1]
> 
> I am sure I still have older users of cerowrt out there flooding
> comcast with dhcpv6 requests from that particular bug, and not
> noticing it.
> 
>> We need to keep
>> heartbeats and management traffic alive to be able to solve these
>> problems.
> 
> Heartbeats and management traffic are naturally sparse, and thus
> automagically prioritized by fq_codel.
> 
> It also uses ECN if you choose, which allows for low latency without
> packet loss on hosts that use ecn by default. The relevant sysctls
> are:
> 
> net.ipv4.tcp_ecn=1
> net.core.default_qdisc=fq_codel
> 
> Which I usually put in a file in /etc/sysctl.d
> 
> and there is also now support for enabling ecn on a per route basis in
> the latest kernels, as people are still dubious about being able to
> use ecn everywhere. But ecn is not required to be fully enabled for
> fq_codel to work it's magic.
> 
> But: I know of at least one large web service provider enabling ecn
> AND fq_codel by default, everywhere now ( archive.org ) and there are
> no doubt countless others.
> 
> In your case, with the 10+GigE hardware queues running mq, you can
> have a top level tc filter to redirect stuff based on markings into
> the "right" hardware queues, and have fq_codel or sch_fq attached
> underneath it for each hardware queue to further manage the traffic.
> The latter part of that happens automagically with the first sysctl
> above, the tc part is kind of expensive to write rules for, I wish we
> had an explicit "diffserv" filter to use instead of one tc rule per
> diffserv codepoint you cared about.
> 
> sch_fq, is a host only solution, mostly targetted at tcp traffic, and
> that *does* support explicit prioritization. It's pretty amazing what
> it does...
> 
> All I have ever argued on these fronts is merely that people try these
> new qdiscs in *their* realistic scenarios and with tools like
> netperf-wrapper, to see if they help or not, and (hopefully) share
> their results so we can further improve matters.
> 
> We are well aware of how much sqm-scripts + fq_codel help cable and
> DSL modems in particular (see for example:
> http://burntchrome.blogspot.com/2014/05/fixing-bufferbloat-on-comcasts-blast.html 
> [2]
> or http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html [3] 
> ),
> 
> and we are working on some derivative ideas to improve wifi as part of
> the make-wifi-fast effort. (
> http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf 
> [4]
> )
> 
> In other news...
> 
> Work is resuming on "sch_cake", which combines htb + fq_codel +
> diffserv (and priority) classification into one qdisc (replacing the
> sqm-scripts entirely) and I do hope we can get some more testers of
> the patches over the next few months. The discussion of "cake" is over
> on the codel list on bufferbloat.net where the current patch set is
> linked to and the flaws discussed. I do hope this qdisc will also
> scale up to 40GigE and be of use in doing traffic engineering at these
> speeds, and PLEASE join us in discussing the needed feature set(s) for
> your use cases.
> 
> And I do generally hope that these algorithms will start appearing the
> basic ethernet and switch hardware at some point soon. I am backing a
> kickstarter project to get it out there in an FPGA at least... (see:
> https://www.kickstarter.com/projects/onetswitch/onetswitch-open-source-hardware-for-networking 
> [5]
> )
> 
>> It seems that fq_codel really helps with latency and
>> bufferbloat (which would also be good for us) and it is possible that
>> it would be adequate since we are DSCP marking packets and the
>> switches are applying L2 COS. I'll have to look over it. If you have
>> some information that would be helpful, I'd be happy to have it.
> 
> Oy. More talks and documentation than you can shake a stick at.
> 
> we have put a collection of the best up here:
> http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos [6]
> 
> I enjoy Stephen Hemminger's presos a lot, although my modena talks go
> into more detail. Van Jacobson's talk is essential, and the shortest
> talk I have ever given on these subjects was the uknof talk.
> 
> Most of my work has been focused on fixing the edge of the internet,
> rather than stuff in the data center, but I am increasingly interested
> in what good all this stuff does in these environments, as I do hope
> to see these algorithms start showing up soon in Load Balancers,
> BRAS's and CMTSes and ISP gear like that.
> 
> Certainly sch_fq is rather widely deployed at at least one large web
> service provider, in particular. :) I don't have any data as to where
> else it is deployed...
> 
> On Wed, Mar 18, 2015 at 4:41 PM, Dave Taht <dave.taht@gmail.com> wrote: 
> On Wed, Mar 18, 2015 at 3:03 PM, Robert LeBlanc <robert@leblanc.us> 
> wrote:
> I can't speak for the specific configuration, but we are having
> challenges using the mqprio qdisc on the Intel XL710 adapters. We have
> engaged Intel, but we are still having trouble. If you want priority
> queuing and using the hardware TX queues, you may have problems with
> this family of cards.
> 
> I am very curious about benchmarks of fq_codel (vs mqprio or
> pfifo_fast) at these speeds on adaptors like this, preferably driven
> by repeatable tests like the rrul test in netperf-wrapper.
> 
> In particular, what happens if only one tx queue is enabled with 
> fq_codel?
> 
> On Wed, Mar 18, 2015 at 1:07 PM, Nieścierowicz Adam
> <adam.niescierowicz@justnet.pl> wrote: Hi,
> 
> in the near future we are planning to purchase new equipment for the 
> router.
> 
> Our requirements are:
> 
> - four or more 1GbE interfaces
> 
> - two-four 10GbE interfaces
> 
> - two 40GbE interfaces
> 
> - multicast routing with throughput input 1-2 Gb/s output 4x1-2Gb/s
> 
> Can someone share your experience what equipment you choose to such a
> configuration
> 
> ---
> Thanks,
> Adam Nieścierowicz
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html [7] 
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html [7]

  --
  Dave Täht
  Let's make wifi fast, less jittery and reliable again!

https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb [8]



Links:
------
[1] http://www.bufferbloat.net/issues/441
[2] 
http://burntchrome.blogspot.com/2014/05/fixing-bufferbloat-on-comcasts-blast.html
[3] http://snapon.lab.bufferbloat.net/~cero2/jimreisert/results.html
[4] 
http://snapon.lab.bufferbloat.net/~d/ieee802.11-sept-17-2014/11-14-1265-00-0wng-More-on-Bufferbloat.pdf
[5] 
https://www.kickstarter.com/projects/onetswitch/onetswitch-open-source-hardware-for-networking
[6] http://www.bufferbloat.net/projects/cerowrt/wiki/Bloat-videos
[7] http://vger.kernel.org/majordomo-info.html
[8] https://plus.google.com/u/0/107942175615993706558/posts/TVX3o84jjmb