Netfilter applied to specific interfaces only

Netfilter applied to specific interfaces only

[Jim Mellander]

Greetings.

In the HPC world, and in network intrusion detection, network
performance is paramount.  We've found that just having the iptables
kernel module loaded without any ruleset substantially reduces
performance at high traffic rates.  Some preliminary performance
measurements:

The total traffic (reported by the iperfs) was:

~18 Gbps with IPTables enabled - no ruleset
~24 Gbps with IPTables disabled

Disabling IPTables (and unloading the associated kernel modules)
seemed to significantly improve performance, but running with IPTables
disabled in production is undesirable.

Typically, we have interfaces that are external facing that we would
like to run IPtables on, but the internal interfaces which are just
for internal cluster communications must run as fast as possible.  A
similar issue occurs during high-speed network intrusion detection -
we want the management interface to be subject to iptables, but we
don't want the performance hit of netfilter impeding traffic at the
interfaces monitoring the network.

So, what would be desirable to see is a sysctl setting that would tell
netfilter to *completely* stay out of the way on a per-interface
basis.  Many supercomputers run linux, and it would be nice to also
run iptables, but the performance hit is unacceptable.

Thanks in advance,

Jim Mellander
NERSC Cybersecurty

Re: Netfilter applied to specific interfaces only

[Jan Engelhardt]

On Friday 2013-03-08 20:14, Jim Mellander wrote:
>
>In the HPC world, and in network intrusion detection, network
>performance is paramount.  We've found that just having the iptables
>kernel module loaded without any ruleset substantially reduces
>performance at high traffic rates.

This one is a known issue with ip_tables/x_tables, and solved in 
xtables2 where you can deallocate the base chains when empty -- (more 
accurately, they do not exist by default and need to be created first) 
-- given finer control over what is being executed.


>Disabling IPTables (and unloading the associated kernel modules)
>seemed to significantly improve performance, but running with IPTables
>disabled in production is undesirable.
>
>Typically, we have interfaces that are external facing that we would
>like to run IPtables on, but the internal interfaces which are just
>for internal cluster communications must run as fast as possible.

Netfilter hooks only run on selected protocols, so at least
the special protocols InfiniBand and what-have-you are unaffected.

Assuming though that you are talking about plain Ethernet protocols:
testing for an interface will be the one costly operation
that all packets have to muster. I suppose the only way to get
around this are network namespaces, as hooks are per netns.
But netns itself probably brings you some new caveats.


Jan

Re: Netfilter applied to specific interfaces only

[Pablo Neira Ayuso]

On Fri, Mar 08, 2013 at 08:52:37PM +0100, Jan Engelhardt wrote:
> 
> On Friday 2013-03-08 20:14, Jim Mellander wrote:
> >
> >In the HPC world, and in network intrusion detection, network
> >performance is paramount.  We've found that just having the iptables
> >kernel module loaded without any ruleset substantially reduces
> >performance at high traffic rates.
> 
> This one is a known issue with ip_tables/x_tables, and solved in 
> xtables2 where you can deallocate the base chains when empty -- (more 
> accurately, they do not exist by default and need to be created first) 
> -- given finer control over what is being executed.

Just for the record: this idea was initially introduced by nftables
back in 2009.

Regards.

Re: Netfilter applied to specific interfaces only

[Jan Engelhardt]

On Saturday 2013-03-09 23:10, Pablo Neira Ayuso wrote:
>On Fri, Mar 08, 2013 at 08:52:37PM +0100, Jan Engelhardt wrote:
>>On Friday 2013-03-08 20:14, Jim Mellander wrote:
>>>
>>>just having the iptables kernel module loaded without any ruleset 
>>>substantially reduces performance at high traffic rates.
>>
>>This one is a known issue with ip_tables/x_tables, and solved in 
>>xtables2 where you can deallocate the base chains when empty -- (more 
>>accurately, they do not exist by default and need to be created first) 
>>-- given finer control over what is being executed.
>
>Just for the record: this idea was initially introduced by nftables 
>back in 2009.

This is now recorded in the xt2 userdoc, where a section has been added 
indicating capability origin by date and thereby implementation.

Re: Netfilter applied to specific interfaces only

[Humberto Jucá]

Hi,

This is an much discussed issue in firewall forums.
I need to study a little more about it, but my current opinion:

1. The servers should not do "any filtering" - except in specific
cases. They should be placed in a DMZ segment or serverfarm. However,
the access to these segments is controlled by a firewall (clustered or
not). So, you can focus on optimizing firewalls.

2. I don't believe there is a native solution for this issue. It's not
for nothing that the business solutions that promise this are the
highest price. This is the kind of solution you'll have to "design" -
If you want a fairly significant performance gain.
- Search for "fpga firewall".
  Something like this: http://netfpga.org/ or
http://www.xilinx.com/products/intellectual-property/TEMAC.htm

- You can also look for solutions based on GPU.
  Take a look at the Suricata project
(http://suricata-ids.org/features/) - Experimental GPU (Unfortunately
it is still experimental).




2013/3/8 Jim Mellander <jmellander@lbl.gov>:
> Greetings.
>
> In the HPC world, and in network intrusion detection, network
> performance is paramount.  We've found that just having the iptables
> kernel module loaded without any ruleset substantially reduces
> performance at high traffic rates.  Some preliminary performance
> measurements:
>
> The total traffic (reported by the iperfs) was:
>
> ~18 Gbps with IPTables enabled - no ruleset
> ~24 Gbps with IPTables disabled
>
> Disabling IPTables (and unloading the associated kernel modules)
> seemed to significantly improve performance, but running with IPTables
> disabled in production is undesirable.
>
> Typically, we have interfaces that are external facing that we would
> like to run IPtables on, but the internal interfaces which are just
> for internal cluster communications must run as fast as possible.  A
> similar issue occurs during high-speed network intrusion detection -
> we want the management interface to be subject to iptables, but we
> don't want the performance hit of netfilter impeding traffic at the
> interfaces monitoring the network.
>
> So, what would be desirable to see is a sysctl setting that would tell
> netfilter to *completely* stay out of the way on a per-interface
> basis.  Many supercomputers run linux, and it would be nice to also
> run iptables, but the performance hit is unacceptable.
>
> Thanks in advance,
>
> Jim Mellander
> NERSC Cybersecurty
> --
> To unsubscribe from this list: send the line "unsubscribe netfilter" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: Netfilter applied to specific interfaces only

[Neal Murphy]

On Sunday, March 10, 2013 10:34:36 AM Humberto Jucá wrote:
> Hi,
> 
> This is an much discussed issue in firewall forums.
> I need to study a little more about it, but my current opinion:
> 
> 1. The servers should not do "any filtering" - except in specific
> cases. They should be placed in a DMZ segment or serverfarm. However,
> the access to these segments is controlled by a firewall (clustered or
> not). So, you can focus on optimizing firewalls.

I humbly disagree. Any server exposed to the internet should be configured to 
limit inbound and outbound access to exactly that which is needed for it to 
operate. For example, an simple web server should allow only new incoming 
conns to ports HTTP and HTTPS from internet; they should block new outgoing 
conns (since a simple web server only serves data over existing conns). 
Management ports, like ssh, should be limited to the least reasonable set of 
addresses expected. Periodic audits should show if these limits have been 
altered. The server is its own first line of defense. The nearest firewall is 
the second line of defense. The perimeter firewall is the last line of 
defense.

Of course, when talking about multi-Gbps links, one needs to install hardware 
that can handle filtering that much data. If the OP has all his inter-LAN 
traffic passing through a firewall, I might suggest his firewall is under-
powered, or might suggest his network topology should be reviewed. If no 
topology changes are possible, then the only recourse is to install a firewall 
that *can* handle filtering that much data.

Re: Netfilter applied to specific interfaces only

[Humberto Jucá]

2013/3/10 Neal Murphy <neal.p.murphy@alum.wpi.edu>:
> I humbly disagree. Any server exposed to the internet should be configured to
> limit inbound and outbound access to exactly that which is needed for it to
> operate. For example, an simple web server should allow only new incoming
> conns to ports HTTP and HTTPS from internet; they should block new outgoing
> conns (since a simple web server only serves data over existing conns).
> Management ports, like ssh, should be limited to the least reasonable set of
> addresses expected. Periodic audits should show if these limits have been
> altered. The server is its own first line of defense. The nearest firewall is
> the second line of defense. The perimeter firewall is the last line of
> defense.

Ok, it's a different approach than I usually take.

IMHO, the first level of security is a firewall and the second a IPS -
so, I can reduce the load processed by IPS (which makes analysis at a
high level - we are working with HP TippingPoint IPS). Servers are
entities that need protecting. We have more than 300 servers. In my
opinion, it would be impractical to manage otherwise. The firewall
rules can't protect a web server of an exploit, but a IPS can.

It is quite likely that the firewall rules on the server does not make
much difference if the service is compromised.
But it's only my opinion.

Re: Netfilter applied to specific interfaces only

[Jim Mellander]

On Sun, Mar 10, 2013 at 7:46 PM, Neal Murphy <neal.p.murphy@alum.wpi.edu> wrote:
> On Sunday, March 10, 2013 10:34:36 AM Humberto Jucá wrote:
>> Hi,
>>
>> This is an much discussed issue in firewall forums.
>> I need to study a little more about it, but my current opinion:
>>
>> 1. The servers should not do "any filtering" - except in specific
>> cases. They should be placed in a DMZ segment or serverfarm. However,
>> the access to these segments is controlled by a firewall (clustered or
>> not). So, you can focus on optimizing firewalls.
>
> I humbly disagree. Any server exposed to the internet should be configured to
> limit inbound and outbound access to exactly that which is needed for it to
> operate. For example, an simple web server should allow only new incoming
> conns to ports HTTP and HTTPS from internet; they should block new outgoing
> conns (since a simple web server only serves data over existing conns).
> Management ports, like ssh, should be limited to the least reasonable set of
> addresses expected. Periodic audits should show if these limits have been
> altered. The server is its own first line of defense. The nearest firewall is
> the second line of defense. The perimeter firewall is the last line of
> defense.
>
> Of course, when talking about multi-Gbps links, one needs to install hardware
> that can handle filtering that much data. If the OP has all his inter-LAN
> traffic passing through a firewall, I might suggest his firewall is under-
> powered, or might suggest his network topology should be reviewed. If no
> topology changes are possible, then the only recourse is to install a firewall
> that *can* handle filtering that much data.
> --

Thanks for all the insightful comments - we will check out xt2.  In
the meantime, we employ tcpwrappers, application access control, etc.
(as well as an advanced IDS: Bro).

A firewall is not appropriate for our environment, at least for the
big iron systems which service a worldwide diverse community, for a
number of reasons:

1. A 100Gps pipe restricts/eliminates firewall choices
2. User community, running a wide range of standard codes, and
homegrown protocols create an impossible task of firewall maintenance
anyway, except for standard ports <1024 & well known high ports that
are used.
3. Basic cutting edge network research occurring - firewalls often
subtly modify the packets (packet reassembly, etc.) and certainly
introduce unacceptable latency.

We certainly believe in the 'belt-and-suspenders' approach to
security, with router ACLs, tcpwrappers, and (in testing) libunaccept
in our arsenal.  It is unfortunate that iptables can't be a part of
this mix (at least for the big iron systems, and our IDS's), but we
look forward to testing xt2 for our purposes.

The NERSC environment is fairly unique, so our solutions are similarly
unique and innovative.

Thanks all,

Jim Mellander
NERSC Cybersecurity

Re: Netfilter applied to specific interfaces only

[Jozsef Kadlecsik]

On Fri, 15 Mar 2013, Jim Mellander wrote:

> On Sun, Mar 10, 2013 at 7:46 PM, Neal Murphy <neal.p.murphy@alum.wpi.edu> wrote:
> > On Sunday, March 10, 2013 10:34:36 AM Humberto Juc? wrote:
> >>
> >> This is an much discussed issue in firewall forums.
> >> I need to study a little more about it, but my current opinion:
> >>
> >> 1. The servers should not do "any filtering" - except in specific
> >> cases. They should be placed in a DMZ segment or serverfarm. However,
> >> the access to these segments is controlled by a firewall (clustered or
> >> not). So, you can focus on optimizing firewalls.
> >
> > I humbly disagree. Any server exposed to the internet should be configured to
> > limit inbound and outbound access to exactly that which is needed for it to
> > operate. For example, an simple web server should allow only new incoming
> > conns to ports HTTP and HTTPS from internet; they should block new outgoing
> > conns (since a simple web server only serves data over existing conns).
> > Management ports, like ssh, should be limited to the least reasonable set of
> > addresses expected. Periodic audits should show if these limits have been
> > altered. The server is its own first line of defense. The nearest firewall is
> > the second line of defense. The perimeter firewall is the last line of
> > defense.
> >
> > Of course, when talking about multi-Gbps links, one needs to install hardware
> > that can handle filtering that much data. If the OP has all his inter-LAN
> > traffic passing through a firewall, I might suggest his firewall is under-
> > powered, or might suggest his network topology should be reviewed. If no
> > topology changes are possible, then the only recourse is to install a firewall
> > that *can* handle filtering that much data.
> > --
> 
> Thanks for all the insightful comments - we will check out xt2.  In
> the meantime, we employ tcpwrappers, application access control, etc.
> (as well as an advanced IDS: Bro).
> 
> A firewall is not appropriate for our environment, at least for the
> big iron systems which service a worldwide diverse community, for a
> number of reasons:
> 
> 1. A 100Gps pipe restricts/eliminates firewall choices
> 2. User community, running a wide range of standard codes, and
> homegrown protocols create an impossible task of firewall maintenance
> anyway, except for standard ports <1024 & well known high ports that
> are used.

ipset may be what can help in these cases.

Best regards,
Jozsef
-
E-mail  : kadlec@blackhole.kfki.hu, kadlecsik.jozsef@wigner.mta.hu
PGP key : http://www.kfki.hu/~kadlec/pgp_public_key.txt
Address : Wigner Research Centre for Physics, Hungarian Academy of Sciences
          H-1525 Budapest 114, POB. 49, Hungary