Recommended hardware for iptables based firewall/router

Recommended hardware for iptables based firewall/router

[Dennis Jacobfeuerborn]

Hi,
we recently bought an Uqbiquity EdgeRouter Pro but it seems the claims
about 2 Mio. pps that it should be able to handle are not real-world
numbers. We are running about 120mbit through this system and are
already seeing the two risc cores struggling with high softirq load and
packet drops.

So my question is what a good hardware base would look like for a linux
based firewall using iptables/conntrack/ipset. Do offload features help
or can't these be used because iptables needs to process the packets
anyway? I assume multiqueuing would be nice too.
The idea is to be able to actually process 1gbit of traffic i.e. handle
two gbit ports (WAN and LAN) at wire-speed.

Does anyone have any specific recommendations for NICs and maybe tips
for other bottlenecks to look out for?

Regards,
  Dennis

Re: Recommended hardware for iptables based firewall/router

[Neal Murphy]

On Saturday, November 01, 2014 11:51:28 PM Dennis Jacobfeuerborn wrote:
> Hi,
> we recently bought an Uqbiquity EdgeRouter Pro but it seems the claims
> about 2 Mio. pps that it should be able to handle are not real-world
> numbers. We are running about 120mbit through this system and are
> already seeing the two risc cores struggling with high softirq load and
> packet drops.
> 
> So my question is what a good hardware base would look like for a linux
> based firewall using iptables/conntrack/ipset. Do offload features help
> or can't these be used because iptables needs to process the packets
> anyway? I assume multiqueuing would be nice too.
> The idea is to be able to actually process 1gbit of traffic i.e. handle
> two gbit ports (WAN and LAN) at wire-speed.
> 
> Does anyone have any specific recommendations for NICs and maybe tips
> for other bottlenecks to look out for?

I've been using a Lanner 7530 for some time now (the 7525 is the current 
'replacement' for it); it runs the recently released Smoothwall 3.1 firewall*. 
Basically, a dual-core 1.6GHz Atom CPU with Intel NICs and 64MiB RAM can 
saturate four gigE links long term using 17-25W.

If you want more than netfilter (such as squid, snort, clamav, et al.), you'll 
want 1-2 GiB RAM and faster CPUs. And maybe more CPUs. If you want VPNs IPSEC 
and/or OpenVPN), you'll need at least faster CPUs.

Offload features usually preclude proper operation of netfilter.

GigE PCI NICs usually top out at 250-350Mb/s (limited by the PCI bus).

Intel NICs are generally the best. (I believe their lineage goes back to DEC, 
which might explain it.) RealTek's offerings of the last five years or so are 
also pretty good.

N



* Proper disclosure dictates that I state: I'm currently the lead dev. for 
Smoothwall Express (GNU/Linux/iptables). I mention it and its capabilities 
solely because I am very familiar with v3.1 and with testing/running it on a 
number of platforms--from a Y2K 600MHz Gateway PIII to an 8CPU/6GiB/virtio KVM 
on a Vishera 8350 with 16GiB RAM.

Re: Recommended hardware for iptables based firewall/router

[Dennis Jacobfeuerborn]

On 02.11.2014 23:38, Neal Murphy wrote:
> On Saturday, November 01, 2014 11:51:28 PM Dennis Jacobfeuerborn wrote:
>> Hi,
>> we recently bought an Uqbiquity EdgeRouter Pro but it seems the claims
>> about 2 Mio. pps that it should be able to handle are not real-world
>> numbers. We are running about 120mbit through this system and are
>> already seeing the two risc cores struggling with high softirq load and
>> packet drops.
>>
>> So my question is what a good hardware base would look like for a linux
>> based firewall using iptables/conntrack/ipset. Do offload features help
>> or can't these be used because iptables needs to process the packets
>> anyway? I assume multiqueuing would be nice too.
>> The idea is to be able to actually process 1gbit of traffic i.e. handle
>> two gbit ports (WAN and LAN) at wire-speed.
>>
>> Does anyone have any specific recommendations for NICs and maybe tips
>> for other bottlenecks to look out for?
> 
> I've been using a Lanner 7530 for some time now (the 7525 is the current 
> 'replacement' for it); it runs the recently released Smoothwall 3.1 firewall*. 
> Basically, a dual-core 1.6GHz Atom CPU with Intel NICs and 64MiB RAM can 
> saturate four gigE links long term using 17-25W.
> 
> If you want more than netfilter (such as squid, snort, clamav, et al.), you'll 
> want 1-2 GiB RAM and faster CPUs. And maybe more CPUs. If you want VPNs IPSEC 
> and/or OpenVPN), you'll need at least faster CPUs.
> 
> Offload features usually preclude proper operation of netfilter.
> 
> GigE PCI NICs usually top out at 250-350Mb/s (limited by the PCI bus).
> 
> Intel NICs are generally the best. (I believe their lineage goes back to DEC, 
> which might explain it.) RealTek's offerings of the last five years or so are 
> also pretty good.

After seeing the EdgeRouter not being able to handle the promised
capacity even remotely I'm a bit suspicious of these "embedded"
solutions. These processors only seem to be able to handle a decent
amount of traffic in a pure routing best case scenario. I get the
impression that as soon as you do add a bit of firewalling they need to
fall back to slow execution paths and then the cpu's are seriously
underpowered.

But then it might just be the EdgeRouter that is especially terrible. We
immediately ran into problems when I found out that they had *reduced*
the size of the connection tracking table to 16k entries from the
default 64k. I cannot imagine why anyone would do this for a system that
is designed to be a firewall.

Regards,
  Dennis

Re: Recommended hardware for iptables based firewall/router

[Yucong Sun]

Dennis Jacobfeuerborn <dennisml@conversis.de>

The EdgeRouter 's asic couldn't handle all use cases ,  Having some
special rule will make it go to "offload" disabled mode.  You should
research if that's the problem.

As for Linux as a router, the key thing you want to test for is PPS,
not BPS.  Commodity hardware should be able to handle up to 1Mpps. Buy
the best Xeon within your budget. Don't bother look at anything else.
(if your project is serious and need to survive a ddos attack)

Cheers.

On Sat, Nov 8, 2014 at 4:48 PM, Yucong Sun <sunyucong@gmail.com> wrote:
>
> The EdgeRouter 's asic couldn't handle all use cases , Having some special
> rule will make it go to "offload" disabled mode.
>
> You should research if that's the problem.
>
> As for Linux as a router, the key thing you want to test for is PPS, not
> BPS. Commodity hardware should be able to handle up to 1Mpps. Buy the best
> Xeon within your budget. Don't bother look at anything else. (if your
> project is serious and need to survive a ddos attack)
>
> Cheers

Re: Recommended hardware for iptables based firewall/router

[Dennis Jacobfeuerborn]

On 09.11.2014 01:49, Yucong Sun wrote:
> Dennis Jacobfeuerborn <dennisml@conversis.de>
> 
> The EdgeRouter 's asic couldn't handle all use cases ,  Having some
> special rule will make it go to "offload" disabled mode.  You should
> research if that's the problem.

Yes that seems to be the problem. Unfortunately the only things we use
are vlan's and iptables+conntrack which I consider the be fairly
standard features required for basic firewalling. I the system cannot
handle traffic at a decent rate with these features than its hardware
seems to be ill-spec'ed for its purpose.

Things got better when I was able to enable vlan offloading...until the
cpu stalled and the system rebooted itself. Apparently the offloading is
unstable.

None of this inspires confidence in a Product that is specifically
advertised as a router/firewall that is sold with 8 Gbit ports and
promises to handle 2 Mio+ pps.

> As for Linux as a router, the key thing you want to test for is PPS,
> not BPS.  Commodity hardware should be able to handle up to 1Mpps. Buy
> the best Xeon within your budget. Don't bother look at anything else.
> (if your project is serious and need to survive a ddos attack)

For now I have chosen a 2 quad-core cpu Xeon system I already have here
and that has multiqueue capable Intel nics and have configured the
appropriate irq affinity and XPS so each queue is handled by a dedicated
core. I think this should provide relatively decent performance.

Regards,
  Dennis

Re: Recommended hardware for iptables based firewall/router

[Stig Thormodsrud]

On 09.11.2014 01:49, Yucong Sun wrote:
> Dennis Jacobfeuerborn <dennisml@conversis.de>
>
> The EdgeRouter 's asic couldn't handle all use cases ,  Having some
> special rule will make it go to "offload" disabled mode.  You should
> research if that's the problem.

I'm a developer on this product.  I'd be interested in the test cases
that it couldn't handle.

Re: Recommended hardware for iptables based firewall/router

[Dennis Jacobfeuerborn]

On 09.11.2014 06:15, Stig Thormodsrud wrote:
> On 09.11.2014 01:49, Yucong Sun wrote:
>> Dennis Jacobfeuerborn <dennisml@conversis.de>
>>
>> The EdgeRouter 's asic couldn't handle all use cases ,  Having some
>> special rule will make it go to "offload" disabled mode.  You should
>> research if that's the problem.
> 
> I'm a developer on this product.  I'd be interested in the test cases
> that it couldn't handle.

This isn't a test case the system is already in live use.

We are only using zone based firewalling, NAT and network/port groups so
basically just iptables+ipset and a couple of vlan interfaces.

In its default configuration both cpus are pegged at 95% soft-irq usage.
Enabling vlan offloading reduces that quite a bit...but apparently make
the system reboot itself about once every two days.

Right at this moment on the wan interface I see 34.273 rx pps and 57.232
tx pps which is going to increase a bit in the coming hours but gives a
good picture about the the system has to handle right now.

Regards,
  Dennis

Re: Recommended hardware for iptables based firewall/router

[Dennis Jacobfeuerborn]

On 09.11.2014 15:05, Dennis Jacobfeuerborn wrote:
> On 09.11.2014 06:15, Stig Thormodsrud wrote:
>> On 09.11.2014 01:49, Yucong Sun wrote:
>>> Dennis Jacobfeuerborn <dennisml@conversis.de>
>>>
>>> The EdgeRouter 's asic couldn't handle all use cases ,  Having some
>>> special rule will make it go to "offload" disabled mode.  You should
>>> research if that's the problem.
>>
>> I'm a developer on this product.  I'd be interested in the test cases
>> that it couldn't handle.
> 
> This isn't a test case the system is already in live use.
> 
> We are only using zone based firewalling, NAT and network/port groups so
> basically just iptables+ipset and a couple of vlan interfaces.
> 
> In its default configuration both cpus are pegged at 95% soft-irq usage.
> Enabling vlan offloading reduces that quite a bit...but apparently make
> the system reboot itself about once every two days.
> 
> Right at this moment on the wan interface I see 34.273 rx pps and 57.232
> tx pps which is going to increase a bit in the coming hours but gives a
> good picture about the the system has to handle right now.
> 
> Regards,
>   Dennis

Here you can find a sanitized version of the config we have running
right now:
http://community.ubnt.com/t5/EdgeMAX/CPUs-show-incredibly-high-softirq-usage/m-p/1079417#U1079829

Regards,
  Dennis