Parts of firewall disappearing under load

Parts of firewall disappearing under load

[Chris]

We've got quite a few heavily loaded boxes (ISP shared servers) which
have firewalls enabled.  The firewalls basically allow certain ports,
block some naughty IPs, and use limit and recent to keep some services
under control.

What we've noticed is that on rare occasions, a box will
firewall itself off from the world.  After a few times of this happening
we found that we could sometimes ssh in from certain IPs, but not
others.  That made me curious, so I did a diff of iptables-save output
with a known good state, and sure enough, the two weren't the same.
Much of the firewall was just missing.  I'm not sure if the firewall
rules it output were actually being used or not.

My only guess is that we are being hit particularly hard that day, and
perhaps a table of IPs is getting too large.

Has anyone seen this before?  There isn't anything in the logs to give
me a clue.  If anyone has pointers on how to further debug this, I would
really appreciate it.  

We are using CentOS, with kernels which at this time are
2.6.18-128.1.10.el5.  If I need to be asking on a CentOS or Redhat
specific list, please let me know.

Thanks,
Chris

RE: Parts of firewall disappearing under load

[Jason Pyeron]

> -----Original Message-----
> From: netfilter-owner@vger.kernel.org 
> [mailto:netfilter-owner@vger.kernel.org] On Behalf Of Chris
> Sent: Tuesday, June 02, 2009 9:52
> To: netfilter@vger.kernel.org
> Subject: Parts of firewall disappearing under load
> 
> We've got quite a few heavily loaded boxes (ISP shared 
> servers) which have firewalls enabled.  The firewalls 
> basically allow certain ports, block some naughty IPs, and 
> use limit and recent to keep some services under control.
> 
> What we've noticed is that on rare occasions, a box will 

Can you make a test case? Does it happen on more than one machine?

> firewall itself off from the world.  After a few times of 
> this happening we found that we could sometimes ssh in from 
> certain IPs, but not others.  That made me curious, so I did 
> a diff of iptables-save output with a known good state, and 
> sure enough, the two weren't the same.
> Much of the firewall was just missing.  I'm not sure if the 
> firewall rules it output were actually being used or not.
> 
> My only guess is that we are being hit particularly hard that 
> day, and perhaps a table of IPs is getting too large.

Can you vouch that there are no hardware issues? (disk, ram, or power?)

> 
> Has anyone seen this before?  There isn't anything in the 
> logs to give me a clue.  If anyone has pointers on how to 
> further debug this, I would really appreciate it.  
> 

Not that issue, but I have seen some very random issues with loaded boxes with
bad power supplies.

> We are using CentOS, with kernels which at this time are 
> 2.6.18-128.1.10.el5.  If I need to be asking on a CentOS or 
> Redhat specific list, please let me know.

Re: Parts of firewall disappearing under load

[Chris]

On Tue, Jun 02, 2009 at 10:10:31AM -0400, Jason Pyeron wrote:
> > We've got quite a few heavily loaded boxes (ISP shared 
> > servers) which have firewalls enabled.  The firewalls 
> > basically allow certain ports, block some naughty IPs, and 
> > use limit and recent to keep some services under control.
> > 
> > What we've noticed is that on rare occasions, a box will 
> 
> Can you make a test case? Does it happen on more than one machine?

That's the tricky part.  It happens maybe once or twice a month, and on
different machines.  I don't know of a way to reproduce it.  Any
pointers on information that would be useful to gather at the time it
happens would be extremely useful, since at this point it is a mystery
to me.


> > firewall itself off from the world.  After a few times of 
> > this happening we found that we could sometimes ssh in from 
> > certain IPs, but not others.  That made me curious, so I did 
> > a diff of iptables-save output with a known good state, and 
> > sure enough, the two weren't the same.
> > Much of the firewall was just missing.  I'm not sure if the 
> > firewall rules it output were actually being used or not.
> > 
> > My only guess is that we are being hit particularly hard that 
> > day, and perhaps a table of IPs is getting too large.
> 
> Can you vouch that there are no hardware issues? (disk, ram, or
> power?)
> 
I guess I wouldn't rule it out, but I think it is fairly unlikely, since
it is happening on boxes that are from different batches and
different places in the data center.

Re: Parts of firewall disappearing under load

[Thomas Jacob]

On Tue, 2009-06-02 at 11:31 -0400, Chris wrote:
> On Tue, Jun 02, 2009 at 10:10:31AM -0400, Jason Pyeron wrote:
> > > We've got quite a few heavily loaded boxes (ISP shared 
> > > servers) which have firewalls enabled.  The firewalls 
> > > basically allow certain ports, block some naughty IPs, and 
> > > use limit and recent to keep some services under control.
> > > 
> > > What we've noticed is that on rare occasions, a box will 
> > 
> > Can you make a test case? Does it happen on more than one machine?
> 
> That's the tricky part.  It happens maybe once or twice a month, and on
> different machines.  I don't know of a way to reproduce it.  Any
> pointers on information that would be useful to gather at the time it
> happens would be extremely useful, since at this point it is a mystery
> to me.

If you can actually see that you have a different active rulesets when
it "works" than when it doesn't work, then your problem most likely is
with the ruleset loading/creation process. I am not aware of any
component of netfilter that can change the ruleset by itself without
user space interaction. Of course various dynamic memory tables can get
exhausted (connection tracking, neighbor caches, routing cache etc), but
when this happens you usually get messages in your kernel log
that clearly say so.

How do you manage your ruleset? Check the logs of that solution....

    Thomas

RE: Parts of firewall disappearing under load

[Jason Pyeron]

FYI, please don't cc me use the list. 

> -----Original Message-----
> From: netfilter-owner@vger.kernel.org 
> [mailto:netfilter-owner@vger.kernel.org] On Behalf Of Thomas Jacob
> Sent: Tuesday, June 02, 2009 11:44
> To: Chris
> Cc: netfilter@vger.kernel.org; Jason Pyeron
> Subject: Re: Parts of firewall disappearing under load
> 
> On Tue, 2009-06-02 at 11:31 -0400, Chris wrote:
> > On Tue, Jun 02, 2009 at 10:10:31AM -0400, Jason Pyeron wrote:
> > > > We've got quite a few heavily loaded boxes (ISP shared
> > > > servers) which have firewalls enabled.  The firewalls basically 
> > > > allow certain ports, block some naughty IPs, and use limit and 
> > > > recent to keep some services under control.
> > > > 
> > > > What we've noticed is that on rare occasions, a box will
> > > 
> > > Can you make a test case? Does it happen on more than one machine?
> > 
> > That's the tricky part.  It happens maybe once or twice a 
> month, and 
> > on different machines.  I don't know of a way to reproduce it.  Any 
> > pointers on information that would be useful to gather at 
> the time it 
> > happens would be extremely useful, since at this point it 
> is a mystery 
> > to me.
> 
> If you can actually see that you have a different active 
> rulesets when it "works" than when it doesn't work, then your 
> problem most likely is with the ruleset loading/creation 
> process. I am not aware of any component of netfilter that 
> can change the ruleset by itself without user space 
> interaction. Of course various dynamic memory tables can get 
> exhausted (connection tracking, neighbor caches, routing 
> cache etc), but when this happens you usually get messages in 
> your kernel log that clearly say so.
> 
> How do you manage your ruleset? Check the logs of that solution....
> 

If each machine is rhel/centos it would be managed by the
/etc/sysconfig/iptables file.

Now there are several utilities which may modify it, but all are user run.

/etc/init.d/iptables is how the file is loaded in to memory.

>     Thomas
> 
> --
> To unsubscribe from this list: send the line "unsubscribe 
> netfilter" in the body of a message to 
> majordomo@vger.kernel.org More majordomo info at  
> http://vger.kernel.org/majordomo-info.html
> 

--
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
-                                                               -
- Jason Pyeron                      PD Inc. http://www.pdinc.us -
- Principal Consultant              10 West 24th Street #100    -
- +1 (443) 269-1555 x333            Baltimore, Maryland 21218   -
-                                                               -
-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
This message is copyright PD Inc, subject to license 20080407P00.

Re: Parts of firewall disappearing under load

[Chris]

On Tue, Jun 02, 2009 at 05:43:49PM +0200, Thomas Jacob wrote:
> If you can actually see that you have a different active rulesets when
> it "works" than when it doesn't work, then your problem most likely is
> with the ruleset loading/creation process. I am not aware of any
> component of netfilter that can change the ruleset by itself without
> user space interaction. Of course various dynamic memory tables can get
> exhausted (connection tracking, neighbor caches, routing cache etc), but
> when this happens you usually get messages in your kernel log
> that clearly say so.
> 
> How do you manage your ruleset? Check the logs of that solution....

Nothing is changing the firewall after it is up, other than the addition
of some deny rules.  Once it is up, it is up.  Nothing would cause the
large number of random differences like what happens here.  We have a
lot of servers running nearly identical rules.  The servers that have
the problem run fine for months at a time too :)

Is it possible that a table used by recent or limit could overrun and cause
this type of behavior without necessarily showing up in the logs?

There is another case I can think of that may or may not be related.  We
lost a fairly large customer once because the firewall would
occasionally stop allowing traffic from his facebook application.  We
studied the tcpdumps for any clue as to what was happening, and could
find nothing.  We started emptying chains until we had a bunch of empty
chains with nothing left but a single ACCEPT for everything.  That still
didn't fix the problem.  We had to remove all the chains, and reload to
get things working again.

I have no idea if these shared servers are suffering the same problem,
but dropping everything in the firewall and reloading does fix it.

The problem I've had in the past is that the servers don't have enough
sense to have this problem during slow times of the day.  I can't leave the
machines in this state long enough to study it very carefully without an
angry mob coming after me.

I can fix the problem, just not tell you how it gets that way.  What I
am really asking is if there is any useful information I could gather
while a machine is down to post to this list so I don't have to ask
people to look into their crystal balls to help diagnose it.  If it is a
bug in code or a strange corner case, I'd like to find out what it is.

Chris