Using Netfilter with high bandwidth

Using Netfilter with high bandwidth

[Julien Vehent]

Hi All,

At work, we're building a new office, and we are considering building our 
own edge firewalls instead of giving bucket loads of money to the big guys. 
We're a Linux shop, so it makes sense to build those new firewall/vpn boxes 
using Linux. But we are concerned about performances and complexity. I make a 
simple diagram of what we want below. We would have a point to point WAN 
connection between the two networks, and then an uplink on each side.

So I figured I would ask the Netfilter heavy users:
  * How much traffic can we expect to route to a decently configured Firewall 
? Can we target 10GBPS with good NICs/CPUs and proper kernel tuning, or is 
that completely out of range ?
  * If I recall correctly, some ISPs are using Linux/Netfilter boxes on their 
network. Do we know the limits of such systems ?
  * Can we consider conntrack and conntrack synchronization between master 
and slave ?
  * What type of network cards will handle 1GBPS and 10GBPS (eventually) ? 
Any recommendation on the hardware ?
  * We are considering starting with a base ubuntu setup and then tuning the 
kernel/system to fit our needs. Some distros are more network oriented than 
others, is there anything that would stand out for our setup ?

Any pointer to tuning/recommendations is more than welcome. If you have 
experience with such a setup but don't want to share publicly, feel free to 
contact me directly.


                          ........... ...... ..........
                       ...      I N T E R N E T       ...
             +--------+..                               .+---------+
        500 MBPS          .............................            |500 MBPS
        UPLINK                                                     |UPLINK
             |                                                     |
        +----+-----------+             1 GBPS WAN        +---------+------+
        |                +------------------------------->                |
        | LAN FIREWALL   |---+                           | DATACENTER FW  
|---+
        +---^+-----------+   |                           +---^+-----------+   
|
            || +-------------+                               || 
+-------------+
            ||                                               ||
            ||                                               ||
            ||1 GBPS LAN                                     ||1 GBPS LAN
            ||                                               ||
            ||                                               ||
          ..+v....                                           |v......
        ..         ..                                       ..        ..
       ..   L A N   ..                                     .. Datacenter.
        .............                                       ...........


Thanks a lot everyone :)

Julien

-- 
Julien Vehent - http://jve.linuxwal.info

Re: Using Netfilter with high bandwidth

[Jan Engelhardt]

On Friday 2012-08-31 21:38, Julien Vehent wrote:

> Hi All,
>
> At work, we're building a new office, and we are considering building our own
> edge firewalls instead of giving bucket loads of money to the big guys. We're a
> Linux shop, so it makes sense to build those new firewall/vpn boxes using
> Linux. But we are concerned about performances and complexity. I make a simple
> diagram of what we want below. We would have a point to point WAN connection
> between the two networks, and then an uplink on each side.
>
> So I figured I would ask the Netfilter heavy users:
> * How much traffic can we expect to route to a decently configured Firewall ?
> Can we target 10GBPS with good NICs/CPUs and proper kernel tuning, or is that
> completely out of range ?
> * If I recall correctly, some ISPs are using Linux/Netfilter boxes on their
> network. Do we know the limits of such systems ?
> * Can we consider conntrack and conntrack synchronization between master and
> slave ?
> * What type of network cards will handle 1GBPS and 10GBPS (eventually) ? Any
> recommendation on the hardware ?

Those with multiqueue. Intel is known to have some offerings, check 
there (I don't have the chip numbers at hand).

> * We are considering starting with a base ubuntu setup and then tuning the
> kernel/system to fit our needs. Some distros are more network oriented than
> others, is there anything that would stand out for our setup ?

openSUSE is the only known one to offer the complete Netfilter package 
spectrum.

Re: Using Netfilter with high bandwidth

[Jesper Dangaard Brouer]

On Sat, 2012-09-01 at 00:39 +0200, Jan Engelhardt wrote:
> On Friday 2012-08-31 21:38, Julien Vehent wrote:
> 
> > Hi All,
> >
> > At work, we're building a new office, and we are considering building our own
> > edge firewalls instead of giving bucket loads of money to the big guys. We're a
> > Linux shop, so it makes sense to build those new firewall/vpn boxes using
> > Linux. But we are concerned about performances and complexity. I make a simple
> > diagram of what we want below. We would have a point to point WAN connection
> > between the two networks, and then an uplink on each side.
> >
> > So I figured I would ask the Netfilter heavy users:
> > * How much traffic can we expect to route to a decently configured Firewall ?
> > Can we target 10GBPS with good NICs/CPUs and proper kernel tuning, or is that
> > completely out of range ?

I did a lot of 10Gbit/s routing testing back in 2009:
http://vger.kernel.org/netconf2009_slides/LinuxCon2009_JesperDangaardBrouer_final.pdf

Which showed that Intels Nehalem microarchitecture, was capable of doing
10Gbit bi-directional routing on Linux.  Combined with multiqueue NICs,
where the Intel 10G NIC were the winning NIC.  (Disclaimer, this testing
were without iptables rules)

Notice this was 2009, based on the first Nehalem arch.  I know that
Sandy bridge will improve performance (due to better handling of
outstanding PCI-transactions) but I have not tested the details.

I'm also eager to test the new Intel CPUs E5-26xx, which have something
called  DDIO (Direct Data I/O).  Where they basically allow the NIC to
map data directly into L3-cache:
 http://www.intel.in/content/www/in/en/io/direct-data-i-o.html


> > * If I recall correctly, some ISPs are using Linux/Netfilter boxes on their
> > network. 

I can confirm that I used to work for an ISP, that still have
Linux/Netfilter boxes, that route and police all their customers
Internet traffic.  Just before I left, we replaced all machines with
Nehalem based machines to prepare for 10G upgrade, but I only a single
machine was deployed with 10Gbit/s NICs while I was still there.


> Do we know the limits of such systems ?

I did some lab tests on 10Gbit/s routing on the new hardware, with
approx 150.000 iptables rules and a corresponding HTB (bandwidth
shaping) tree.  We ran into some limits around 4.5Gbit/s, but that was
due to the HTB tree, because it causes serialization the traffic control
layer when transmitting/queueing packets.

Notice that you really have to careful how you structure your ruleset,
if you want this many rules:
http://www.slideshare.net/brouer/netfilter-making-large-iptables-rulesets-scale


> > * Can we consider conntrack and conntrack synchronization between master and
> > slave ?

Never played with contrackd.  Perhaps someone could share their
experience in this area?


> > * What type of network cards will handle 1GBPS and 10GBPS (eventually) ? Any
> > recommendation on the hardware ?
>
> Those with multiqueue. Intel is known to have some offerings, check 
> there (I don't have the chip numbers at hand).

The Intel chip number for the 1Gbit/s NIC is 82576 and for the 10Gbit/s
NIC is 82599.



> > * We are considering starting with a base ubuntu setup and then tuning the
> > kernel/system to fit our needs. Some distros are more network oriented than
> > others, is there anything that would stand out for our setup ?

If you plan to manage the server yourself, I really recommend you just
choose your favorite Linux distro on a standard server.  I have spend
too much time on getting stuff to work on minimal semi-homebrewed
distributions running on flash or disk-drives.

And please remember to increase the number of conntrack entries.

E.g.:
 /proc/sys/net/netfilter/nf_conntrack_max = 900000

And to the calc: Conntrack element size 228 bytes found
in /proc/slabinfo: "nf_conntrack <objsize> = 228 "

 228 * 900000 / 10^6 = 205.2 MB

You should also change the nf_conntrack hash bucket size, as just
increasing the number of conntrack entries, will cause more collisions. 

This is done either when loading the module:
  modprobe nf_conntrack hashsize=300000
Or runtime via /sys:
  echo 300000 > /sys/module/nf_conntrack/parameters/hashsize


Do you plan to write the iptables rules manually in a script, or do you
plan to use a GUI for config?
(I'm just asking because I don't know if there is some good free GUIs
out-there... I once approx 2001 used fwbuilder.org, for a system that
someone else had to admin, they seemed happy...)

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Using Netfilter with high bandwidth

[Julien Vehent]

Thank you Jan & Jesper for the very detailled answers. We have started 
working on a prototype using Intel 10G cards (82599ES). I'll spend some time 
reading all the papers you posted, and will try to write something up at the 
end.

The cost difference compared to vendor prices is just ridiculous. We ca spec 
a 4*10G NICs with dual 8core for ~$6k each, while the equivalent from the big 
guys would be around ~$60/80k a piece.

-- 
Julien Vehent - http://jve.linuxwal.info


On 2012-09-03 03:56, Jesper Dangaard Brouer wrote:
> On Sat, 2012-09-01 at 00:39 +0200, Jan Engelhardt wrote:
>> On Friday 2012-08-31 21:38, Julien Vehent wrote:
>>
>> > Hi All,
>> >
>> > At work, we're building a new office, and we are considering building 
>> our own
>> > edge firewalls instead of giving bucket loads of money to the big guys. 
>> We're a
>> > Linux shop, so it makes sense to build those new firewall/vpn boxes 
>> using
>> > Linux. But we are concerned about performances and complexity. I make a 
>> simple
>> > diagram of what we want below. We would have a point to point WAN 
>> connection
>> > between the two networks, and then an uplink on each side.
>> >
>> > So I figured I would ask the Netfilter heavy users:
>> > * How much traffic can we expect to route to a decently configured 
>> Firewall ?
>> > Can we target 10GBPS with good NICs/CPUs and proper kernel tuning, or 
>> is that
>> > completely out of range ?
>
> I did a lot of 10Gbit/s routing testing back in 2009:
> 
> http://vger.kernel.org/netconf2009_slides/LinuxCon2009_JesperDangaardBrouer_final.pdf
>
> Which showed that Intels Nehalem microarchitecture, was capable of doing
> 10Gbit bi-directional routing on Linux.  Combined with multiqueue NICs,
> where the Intel 10G NIC were the winning NIC.  (Disclaimer, this testing
> were without iptables rules)
>
> Notice this was 2009, based on the first Nehalem arch.  I know that
> Sandy bridge will improve performance (due to better handling of
> outstanding PCI-transactions) but I have not tested the details.
>
> I'm also eager to test the new Intel CPUs E5-26xx, which have something
> called  DDIO (Direct Data I/O).  Where they basically allow the NIC to
> map data directly into L3-cache:
>  http://www.intel.in/content/www/in/en/io/direct-data-i-o.html
>
>
>> > * If I recall correctly, some ISPs are using Linux/Netfilter boxes on 
>> their
>> > network.
>
> I can confirm that I used to work for an ISP, that still have
> Linux/Netfilter boxes, that route and police all their customers
> Internet traffic.  Just before I left, we replaced all machines with
> Nehalem based machines to prepare for 10G upgrade, but I only a single
> machine was deployed with 10Gbit/s NICs while I was still there.
>
>
>> Do we know the limits of such systems ?
>
> I did some lab tests on 10Gbit/s routing on the new hardware, with
> approx 150.000 iptables rules and a corresponding HTB (bandwidth
> shaping) tree.  We ran into some limits around 4.5Gbit/s, but that was
> due to the HTB tree, because it causes serialization the traffic control
> layer when transmitting/queueing packets.
>
> Notice that you really have to careful how you structure your ruleset,
> if you want this many rules:
> 
> http://www.slideshare.net/brouer/netfilter-making-large-iptables-rulesets-scale
>
>
>> > * Can we consider conntrack and conntrack synchronization between 
>> master and
>> > slave ?
>
> Never played with contrackd.  Perhaps someone could share their
> experience in this area?
>
>
>> > * What type of network cards will handle 1GBPS and 10GBPS (eventually) 
>> ? Any
>> > recommendation on the hardware ?
>>
>> Those with multiqueue. Intel is known to have some offerings, check
>> there (I don't have the chip numbers at hand).
>
> The Intel chip number for the 1Gbit/s NIC is 82576 and for the 10Gbit/s
> NIC is 82599.
>
>
>
>> > * We are considering starting with a base ubuntu setup and then tuning 
>> the
>> > kernel/system to fit our needs. Some distros are more network oriented 
>> than
>> > others, is there anything that would stand out for our setup ?
>
> If you plan to manage the server yourself, I really recommend you just
> choose your favorite Linux distro on a standard server.  I have spend
> too much time on getting stuff to work on minimal semi-homebrewed
> distributions running on flash or disk-drives.
>
> And please remember to increase the number of conntrack entries.
>
> E.g.:
>  /proc/sys/net/netfilter/nf_conntrack_max = 900000
>
> And to the calc: Conntrack element size 228 bytes found
> in /proc/slabinfo: "nf_conntrack <objsize> = 228 "
>
>  228 * 900000 / 10^6 = 205.2 MB
>
> You should also change the nf_conntrack hash bucket size, as just
> increasing the number of conntrack entries, will cause more collisions.
>
> This is done either when loading the module:
>   modprobe nf_conntrack hashsize=300000
> Or runtime via /sys:
>   echo 300000 > /sys/module/nf_conntrack/parameters/hashsize
>
>
> Do you plan to write the iptables rules manually in a script, or do you
> plan to use a GUI for config?
> (I'm just asking because I don't know if there is some good free GUIs
> out-there... I once approx 2001 used fwbuilder.org, for a system that
> someone else had to admin, they seemed happy...)

Re: Using Netfilter with high bandwidth

[Luigi Rizzo]

On Mon, Sep 03, 2012 at 09:56:20AM +0200, Jesper Dangaard Brouer wrote:
> On Sat, 2012-09-01 at 00:39 +0200, Jan Engelhardt wrote:
> > On Friday 2012-08-31 21:38, Julien Vehent wrote:
...
> > > * How much traffic can we expect to route to a decently configured Firewall ?
> > > Can we target 10GBPS with good NICs/CPUs and proper kernel tuning, or is that
> > > completely out of range ?
>
> I did a lot of 10Gbit/s routing testing back in 2009:
> http://vger.kernel.org/netconf2009_slides/LinuxCon2009_JesperDangaardBrouer_final.pdf
> 
> Which showed that Intels Nehalem microarchitecture, was capable of doing
> 10Gbit bi-directional routing on Linux.  Combined with multiqueue NICs,
> where the Intel 10G NIC were the winning NIC.  (Disclaimer, this testing
> were without iptables rules)
> 

Jesper's report is extremely interesting but covers routing.
For firewalls, it might be interesting to have a look at the
netmap+ipfw integration

	http://info.iet.unipi.it/~luigi/dummynet/#8696

which i recently completed and runs on linux as well (uses a kernel
module for network I/O, and a userspace versions of ipfw and dummynet
for the firewalling, scheduling, shaping).
I measured over 6 million packets per second (Mpps) with simple
rulesets, and over 2.2 Mpps through dummynet pipes.
This is with 1 core doing the processing.

The links from there should bring you to the other relevant
pieces, but just in case

    http://info.iet.unipi.it/~luigi/netmap
	is the netmap API, drivers etc

    http://info.iet.unipi.it/~luigi/vale
	describes the VALE soft switches, useful for testing

    http://info.iet.unipi.it/~luigi/dummynet
	features of dummynet and ipfw

cheers
luigi

Re: Using Netfilter with high bandwidth

[Jan Engelhardt]

On Thursday 2012-09-06 19:56, Julien Vehent wrote:

> Thank you Jan & Jesper for the very detailled answers. We have started working
> on a prototype using Intel 10G cards (82599ES). I'll spend some time reading
> all the papers you posted, and will try to write something up at the end.
>
> The cost difference compared to vendor prices is just ridiculous. We ca spec a
> 4*10G NICs with dual 8core for ~$6k each, while the equivalent from the big
> guys would be around ~$60/80k a piece.

Well Intel is no poster-boy either. When looking into the home consumer 
section:

Intel Gigabit CT Desktop EXPI9301CT selling for €23.79, a TP-LINK 
TG-3468 for as low as €10.99 at a selected internet shop (but the ratio 
is quite consistent across different shops).

Re: Using Netfilter with high bandwidth

[Marco Padovan]

Solutions like these:

http://shader.kaist.edu/packetshader/

are surfacing lately... and those can't be compared with CPU processing ;)

Il 31/08/2012 21:38, Julien Vehent ha scritto:
> Hi All,
>
> At work, we're building a new office, and we are considering building
> our own edge firewalls instead of giving bucket loads of money to the
> big guys. We're a Linux shop, so it makes sense to build those new
> firewall/vpn boxes using Linux. But we are concerned about
> performances and complexity. I make a simple diagram of what we want
> below. We would have a point to point WAN connection between the two
> networks, and then an uplink on each side.
>
> So I figured I would ask the Netfilter heavy users:
>  * How much traffic can we expect to route to a decently configured
> Firewall ? Can we target 10GBPS with good NICs/CPUs and proper kernel
> tuning, or is that completely out of range ?
>  * If I recall correctly, some ISPs are using Linux/Netfilter boxes on
> their network. Do we know the limits of such systems ?
>  * Can we consider conntrack and conntrack synchronization between
> master and slave ?
>  * What type of network cards will handle 1GBPS and 10GBPS
> (eventually) ? Any recommendation on the hardware ?
>  * We are considering starting with a base ubuntu setup and then
> tuning the kernel/system to fit our needs. Some distros are more
> network oriented than others, is there anything that would stand out
> for our setup ?
>
> Any pointer to tuning/recommendations is more than welcome. If you
> have experience with such a setup but don't want to share publicly,
> feel free to contact me directly.
>
>
>                          ........... ...... ..........
>                       ...      I N T E R N E T       ...
>             +--------+..                               .+---------+
>        500 MBPS          .............................            |500
> MBPS
>        UPLINK                                                     |UPLINK
>             |                                                     |
>        +----+-----------+             1 GBPS WAN       
> +---------+------+
>        |               
> +------------------------------->                |
>        | LAN FIREWALL   |---+                           | DATACENTER
> FW  |---+
>        +---^+-----------+   |                          
> +---^+-----------+   |
>            || +-------------+                               ||
> +-------------+
>            ||                                               ||
>            ||                                               ||
>            ||1 GBPS LAN                                     ||1 GBPS LAN
>            ||                                               ||
>            ||                                               ||
>          ..+v....                                           |v......
>        ..         ..                                       ..        ..
>       ..   L A N   ..                                     .. Datacenter.
>        .............                                       ...........
>
>
> Thanks a lot everyone :)
>
> Julien
>

Re: Using Netfilter with high bandwidth

[Jan Engelhardt]

On Saturday 2012-09-01 00:39, Jan Engelhardt wrote:
>On Friday 2012-08-31 21:38, Julien Vehent wrote:
>
>>At work, we're building a new office, and we are considering
>>building our own edge firewalls instead of giving bucket loads of
>>money to the big guys. We're a Linux shop, so it makes sense to
>>build those new firewall/vpn boxes using Linux. But we are
>>concerned about performances and complexity. I make a simple
>>diagram of what we want below. We would have a point to point WAN
>>connection between the two networks, and then an uplink on each
>>side.
>>
>>[...]
>>* What type of network cards will handle 1GBPS and 10GBPS
>>(eventually) ? Any recommendation on the hardware ?
>
>Those with multiqueue. Intel is known to have some offerings, check
>there (I don't have the chip numbers at hand).

The chip/card I was thinking of (lspci output):

02:00.0 Ethernet controller [0200]:
Broadcom Corporation NetXtreme II BCM5716 Gigabit Ethernet [14e4:163b] (rev 20)
        Subsystem: Dell Device [1028:02a5]
                Product Name: Broadcom NetXtreme II Ethernet Controller
                Read-only fields:
                        [PN] Part number: BCM95716C1
                        [V0] Vendor specific: 5.0.13

This card has 8 queues according to /proc/interrupts.

>> kernel/system to fit our needs. Some distros are more network oriented than
>> others, is there anything that would stand out for our setup ?
>
>openSUSE is the only known one to offer the complete Netfilter package 
>spectrum.