Best qdisc for interfaces of a firewall?

Best qdisc for interfaces of a firewall?

[Dennis Jacobfeuerborn]

Hi,
I just looked at the interfaces of our EdgeRouter Pro appliance that we
plan to replace (due to it apparently being overloaded at 150Mbit) and
see that they all have a qdisc of "noqueue".

What is the best qdisc to select for a pure firewall system? I can't
find any decent information about the various qdiscs and which to chose
in specific situations. For example there seems to exist a multiq
scheduler but I cannot find a lot of information about its
characteristics plus I already assigned the irq of each queue of the nic
to individual cores so I wonder if something like multiq is even necessary.

I'm also wondering about fairness and if that might be a legitimate
reason to chose somehting like noqueue so one flooding flow cannot hog
the queue and penalize all other flows.

Any ideas what would be a well performing yet fair choice here?

Regards,
  Dennis

Re: Best qdisc for interfaces of a firewall?

[Dennis Jacobfeuerborn]

The firmware is the current 1.5 release (well current before the very
recent 1.6 one) so it's not really old.

fq_codel is not in use and all interface use a noqueue qdisc.

We are only using zone based firewalling, NAT and network/port groups so
basically just iptables+ipset and a couple of vlan interfaces.

In its default configuration both cpus are pegged at 95% soft-irq usage.
Enabling vlan offloading reduces that quite a bit...but apparently make
the system reboot itself about once every two days.

On 09.11.2014 07:29, josh Reynolds wrote:
> There is an issue on older firmware with edgerouterand fq_codel, Dave
> would be the one to talk about that.. it's a codel/kernel thing though.
> 
> I know wisps running full line rate and thousands of customers through
> edgerouter pros with no problems. What are you having issues with?
> 
> On 11/08/2014 03:57 PM, Dennis Jacobfeuerborn wrote:
>> Hi,
>> I just looked at the interfaces of our EdgeRouter Pro appliance that we
>> plan to replace (due to it apparently being overloaded at 150Mbit) and
>> see that they all have a qdisc of "noqueue".
>>
>> What is the best qdisc to select for a pure firewall system? I can't
>> find any decent information about the various qdiscs and which to chose
>> in specific situations. For example there seems to exist a multiq
>> scheduler but I cannot find a lot of information about its
>> characteristics plus I already assigned the irq of each queue of the nic
>> to individual cores so I wonder if something like multiq is even
>> necessary.
>>
>> I'm also wondering about fairness and if that might be a legitimate
>> reason to chose somehting like noqueue so one flooding flow cannot hog
>> the queue and penalize all other flows.
>>
>> Any ideas what would be a well performing yet fair choice here?
>>
>> Regards,
>>    Dennis
>> -- 
>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
> 
> 

Re: Best qdisc for interfaces of a firewall?

[Alan Goodman]

Surely pfifo is the least resource intensive?  If the device is already 
overloaded then I could not recommend any qdisc that does additional 
processing...

Alan

On 09/11/14 13:58, Dennis Jacobfeuerborn wrote:
> The firmware is the current 1.5 release (well current before the very
> recent 1.6 one) so it's not really old.
>
> fq_codel is not in use and all interface use a noqueue qdisc.
>
> We are only using zone based firewalling, NAT and network/port groups so
> basically just iptables+ipset and a couple of vlan interfaces.
>
> In its default configuration both cpus are pegged at 95% soft-irq usage.
> Enabling vlan offloading reduces that quite a bit...but apparently make
> the system reboot itself about once every two days.
>
> On 09.11.2014 07:29, josh Reynolds wrote:
>> There is an issue on older firmware with edgerouterand fq_codel, Dave
>> would be the one to talk about that.. it's a codel/kernel thing though.
>>
>> I know wisps running full line rate and thousands of customers through
>> edgerouter pros with no problems. What are you having issues with?
>>
>> On 11/08/2014 03:57 PM, Dennis Jacobfeuerborn wrote:
>>> Hi,
>>> I just looked at the interfaces of our EdgeRouter Pro appliance that we
>>> plan to replace (due to it apparently being overloaded at 150Mbit) and
>>> see that they all have a qdisc of "noqueue".
>>>
>>> What is the best qdisc to select for a pure firewall system? I can't
>>> find any decent information about the various qdiscs and which to chose
>>> in specific situations. For example there seems to exist a multiq
>>> scheduler but I cannot find a lot of information about its
>>> characteristics plus I already assigned the irq of each queue of the nic
>>> to individual cores so I wonder if something like multiq is even
>>> necessary.
>>>
>>> I'm also wondering about fairness and if that might be a legitimate
>>> reason to chose somehting like noqueue so one flooding flow cannot hog
>>> the queue and penalize all other flows.
>>>
>>> Any ideas what would be a well performing yet fair choice here?
>>>
>>> Regards,
>>>     Dennis
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>

Re: Best qdisc for interfaces of a firewall?

[josh Reynolds]

I would suggest talking to stig and an-cheng via either the ubnt forum,
or contact them in ##ubnt on freenode irc.

Rebooting like that is not normal behavior.

The only thing I can think of is maybe you have some very inefficient
firewall rules that could be cleaned up a bit? Do you have any
"external" packages installed that aren't natively on the router?

On 11/09/2014 04:58 AM, Dennis Jacobfeuerborn wrote:
> The firmware is the current 1.5 release (well current before the very
> recent 1.6 one) so it's not really old.
>
> fq_codel is not in use and all interface use a noqueue qdisc.
>
> We are only using zone based firewalling, NAT and network/port groups so
> basically just iptables+ipset and a couple of vlan interfaces.
>
> In its default configuration both cpus are pegged at 95% soft-irq usage.
> Enabling vlan offloading reduces that quite a bit...but apparently make
> the system reboot itself about once every two days.
>
> On 09.11.2014 07:29, josh Reynolds wrote:
>> There is an issue on older firmware with edgerouterand fq_codel, Dave
>> would be the one to talk about that.. it's a codel/kernel thing though.
>>
>> I know wisps running full line rate and thousands of customers through
>> edgerouter pros with no problems. What are you having issues with?
>>
>> On 11/08/2014 03:57 PM, Dennis Jacobfeuerborn wrote:
>>> Hi,
>>> I just looked at the interfaces of our EdgeRouter Pro appliance that we
>>> plan to replace (due to it apparently being overloaded at 150Mbit) and
>>> see that they all have a qdisc of "noqueue".
>>>
>>> What is the best qdisc to select for a pure firewall system? I can't
>>> find any decent information about the various qdiscs and which to chose
>>> in specific situations. For example there seems to exist a multiq
>>> scheduler but I cannot find a lot of information about its
>>> characteristics plus I already assigned the irq of each queue of the nic
>>> to individual cores so I wonder if something like multiq is even
>>> necessary.
>>>
>>> I'm also wondering about fairness and if that might be a legitimate
>>> reason to chose somehting like noqueue so one flooding flow cannot hog
>>> the queue and penalize all other flows.
>>>
>>> Any ideas what would be a well performing yet fair choice here?
>>>
>>> Regards,
>>>     Dennis
>>> -- 
>>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>

Re: Best qdisc for interfaces of a firewall?

[Rick Jones]

On 11/09/2014 05:58 AM, Dennis Jacobfeuerborn wrote:
> In its default configuration both cpus are pegged at 95% soft-irq usage.
> Enabling vlan offloading reduces that quite a bit...but apparently make
> the system reboot itself about once every two days.

Someone else has already commented about asking someone about the 
reboots.  I'll simply mention the "perf" utility and assuming the CPU 
type in your device supports it you could take some profiles to see just 
where the time is being spent.  You can probably guess already that it 
involves vlan processing but it may be helpful to have more of the details.

rick jones

Re: Best qdisc for interfaces of a firewall?

[Dave Taht]

Several notes on this thread:

1) Inefficient firewall rules are a common, and it is often possible
to seriously optimize them. Find a place to post them so others can
take a look.

But a huge problem on devices with offloads is that the first iptables
rule you insert costs quite a lot if it causes offloads to be
disabled.

2) fq_codel is very efficient and rarely shows up as a significant
fraction of cpu - other overheads (like firewall rules and HTB)
dominate. If you are trying to push more than 60Mbit through HTB + any
qdisc on the edgerouter lite we hit problems.

3) We (meaning cerowrt and openwrt and many others) worked very hard
on making 3.10.12 and later the best possible kernel for routers we
could make, and I do hope that the latest 1.6 edgerouter release
proves stabler and faster than the last. But see points 1 and 2.

4) At this point in time I am intensely frustrated with all the
hardware offloading based products. In every case they work just fine
in benchmarks, but often fall appallingly short of their rated specs
when some more real-world configuration is used. And furthermore, the
chipmakers with their "secret sauce" in their firmware are generally
unwilling to open that up so that it could be improved to match the
requirements of the real world.

So in a quest to get high speed (gigE+) packet forwarding with
sufficient intelligence, I've been mostly mucking with rangeley and
parallela hardware of late, and have given up entirely on the latest
generation of hardware-with-offloads-with-closed-source-firmware.

On Sun, Nov 9, 2014 at 5:58 AM, Dennis Jacobfeuerborn
<dennisml@conversis.de> wrote:
> The firmware is the current 1.5 release (well current before the very
> recent 1.6 one) so it's not really old.
>
> fq_codel is not in use and all interface use a noqueue qdisc.
>
> We are only using zone based firewalling, NAT and network/port groups so
> basically just iptables+ipset and a couple of vlan interfaces.
>
> In its default configuration both cpus are pegged at 95% soft-irq usage.
> Enabling vlan offloading reduces that quite a bit...but apparently make
> the system reboot itself about once every two days.
>
> On 09.11.2014 07:29, josh Reynolds wrote:
>> There is an issue on older firmware with edgerouterand fq_codel, Dave
>> would be the one to talk about that.. it's a codel/kernel thing though.
>>
>> I know wisps running full line rate and thousands of customers through
>> edgerouter pros with no problems. What are you having issues with?
>>
>> On 11/08/2014 03:57 PM, Dennis Jacobfeuerborn wrote:
>>> Hi,
>>> I just looked at the interfaces of our EdgeRouter Pro appliance that we
>>> plan to replace (due to it apparently being overloaded at 150Mbit) and
>>> see that they all have a qdisc of "noqueue".
>>>
>>> What is the best qdisc to select for a pure firewall system? I can't
>>> find any decent information about the various qdiscs and which to chose
>>> in specific situations. For example there seems to exist a multiq
>>> scheduler but I cannot find a lot of information about its
>>> characteristics plus I already assigned the irq of each queue of the nic
>>> to individual cores so I wonder if something like multiq is even
>>> necessary.
>>>
>>> I'm also wondering about fairness and if that might be a legitimate
>>> reason to chose somehting like noqueue so one flooding flow cannot hog
>>> the queue and penalize all other flows.
>>>
>>> Any ideas what would be a well performing yet fair choice here?
>>>
>>> Regards,
>>>    Dennis
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe lartc" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>
>>
>
> --
> To unsubscribe from this list: send the line "unsubscribe lartc" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Dave Täht

thttp://www.bufferbloat.net/projects/bloat/wiki/Upcoming_Talks

Re: Best qdisc for interfaces of a firewall?

[Dennis Jacobfeuerborn]

On 10.11.2014 19:04, Dave Taht wrote:
> Several notes on this thread:
> 
> 1) Inefficient firewall rules are a common, and it is often possible
> to seriously optimize them. Find a place to post them so others can
> take a look.
> 
> But a huge problem on devices with offloads is that the first iptables
> rule you insert costs quite a lot if it causes offloads to be
> disabled.

The rules are reasonably efficient. Its the standard EdgeOS zone
configuration where traffic is routed into a chain based on the output
interface and from there into a chain based on the incoming interface.
There the first rule is to accept all related and established traffic
followed by the "real" rules that allow specific traffic based on
addresses, ports and ipsets.

There are some standard Vyatta hook chains involved that are empty but
if these decimate the packet throughput this much then no optimization
can bring the processing to acceptable levels.

> 2) fq_codel is very efficient and rarely shows up as a significant
> fraction of cpu - other overheads (like firewall rules and HTB)
> dominate. If you are trying to push more than 60Mbit through HTB + any
> qdisc on the edgerouter lite we hit problems.
> 
> 3) We (meaning cerowrt and openwrt and many others) worked very hard
> on making 3.10.12 and later the best possible kernel for routers we
> could make, and I do hope that the latest 1.6 edgerouter release
> proves stabler and faster than the last. But see points 1 and 2.

I've been thinking about using 3.16 on the new box since 3.15 apparently
brought some scalability improvements for connection tracking and 3.16
is going to be supported by Ubuntu at least into 2016 makeing it
effectively a long-term supported kernel.

> 4) At this point in time I am intensely frustrated with all the
> hardware offloading based products. In every case they work just fine
> in benchmarks, but often fall appallingly short of their rated specs
> when some more real-world configuration is used. And furthermore, the
> chipmakers with their "secret sauce" in their firmware are generally
> unwilling to open that up so that it could be improved to match the
> requirements of the real world.

That has been my experience as well and it is indeed extremely frustrating.
Since the trouble with the EdgeRouter Pro was causing serious issues I
replaced it with a regular System we had lying around with 8 cores and
multiqueue capable network interfaces. I performed some basic
optimizations like irq affinity and XPS assignment and imported the
ipsets and iptables rules from the EdgeRouter without any changes. At
peak times the cpu utilization shows each core 98-99% idle.

Regards,
  Dennis

Re: Best qdisc for interfaces of a firewall?

[Stig Thormodsrud]

As I mentioned on netfilter 150Mb really shouldn't be a problem with
ER-Pro even without hw offload.  I tried to boot the config you
pointed me to but it's been so obfuscated that I can't boot it.  Would
you mind privately emailing me your config file?