Re: Parts of firewall disappearing under load

[Chris <chris@deksai.com>]

On Tue, Jun 02, 2009 at 10:10:31AM -0400, Jason Pyeron wrote:
> > We've got quite a few heavily loaded boxes (ISP shared 
> > servers) which have firewalls enabled.  The firewalls 
> > basically allow certain ports, block some naughty IPs, and 
> > use limit and recent to keep some services under control.
> > 
> > What we've noticed is that on rare occasions, a box will 
> 
> Can you make a test case? Does it happen on more than one machine?

That's the tricky part.  It happens maybe once or twice a month, and on
different machines.  I don't know of a way to reproduce it.  Any
pointers on information that would be useful to gather at the time it
happens would be extremely useful, since at this point it is a mystery
to me.


> > firewall itself off from the world.  After a few times of 
> > this happening we found that we could sometimes ssh in from 
> > certain IPs, but not others.  That made me curious, so I did 
> > a diff of iptables-save output with a known good state, and 
> > sure enough, the two weren't the same.
> > Much of the firewall was just missing.  I'm not sure if the 
> > firewall rules it output were actually being used or not.
> > 
> > My only guess is that we are being hit particularly hard that 
> > day, and perhaps a table of IPs is getting too large.
> 
> Can you vouch that there are no hardware issues? (disk, ram, or
> power?)
> 
I guess I wouldn't rule it out, but I think it is fairly unlikely, since
it is happening on boxes that are from different batches and
different places in the data center.