Troubleshooting stability issue

Troubleshooting stability issue

[Simon Waters]

One of our firewalls appears to be unstable.

The problems started with adding ip_conntrack and ip_conntrack_ftp, as we need 
to support active mode FTP from inside to the outside.

Prior to this we were using it as a basic port filter (poor iptables), and so 
many of the rules exist to allow return packets without connection tracking.

As such it makes sense for us to "clean-up" the ruleset, but this hasn't 
happened yet.

Due to some interesting (and maybe related?) hardware issues, we've run the 
same firewall configuration on three different servers (all x86, Redhat and 
Debian, IDE and SCSI), including one with the latest (for Debian) 2.6.8 
kernel, and are fairly confident we see the same software issue on all three 
boxes.

Symptom is that by the time we get to it, the box is totally unresponsive to 
local console, is not forwarding packets. In a word "hung".

Memory isn't obviously leaking.

The number of lines in "ip_conntrack" does appear to grow with time, but is 
still way below (at around 3000) the maximum allowed of 32,000+, and isn't 
growing monotonically. I think there may be clues here, if only to what is 
wrong with the ruleset.

Apart from sshd, there is practically nothing running.
atd
crond
inetd (this has nothing configured in /etc/inetd.conf, so I'll remove it).
Postfix is listening on 127.0.0.1:25 in case anything local suddenly needs to 
report anything to me.
lpd (lpd was running and not listening on any ports, so I'll remove it).

Just looking for some helpful pointers on how to investigate this issue 
further.

As even with a "suboptimal" rule set I wouldn't expect the box to hang.
Logs have no useful entries (certainly no "table full" messages).

Don't want to post the full ruleset here, at least not till I've been over it 
with a finetoothed comb. And it is about 230 lines. On the upside quite a lot 
of it has just been obseleted by us relocating all our machines to one site, 
so I can shortly remove large chunks of it.

 Simon


The box is currently running Debian Sarge, with stock kernel;

Linux version 2.4.27-2-386 (horms@tabatha.lab.ultramonkey.org) (gcc version 
3.3.5 (Debian 1:3.3.5-12)) #1 Mon May 16 16:47:51 JST 2005

Module                  Size  Used by    Not tainted
ipt_REJECT              3160   5  (autoclean)
ipt_state                504   2  (autoclean)
iptable_filter          1644   1  (autoclean)
ip_conntrack_ftp        3440   0  (unused)
ip_conntrack           17000   1  [ipt_state ip_conntrack_ftp]
ip_tables              10400   3  [ipt_REJECT ipt_state iptable_filter]
i810_rng                2368   0  (unused)
ehci-hcd               14764   0  (unused)
usb-uhci               19504   0  (unused)
usbcore                52268   1  [ehci-hcd usb-uhci]
e1000                  57676   2
sr_mod                 11640   0  (unused)
scsi_mod               86052   1  [sr_mod]
ide-cd                 27072   0
cdrom                  26212   0  [sr_mod ide-cd]
rtc                     5768   0  (autoclean)
reiserfs              152944   1  (autoclean)
ext3                   65388   0  (autoclean)
jbd                    34628   0  (autoclean) [ext3]
ide-detect               288   0  (autoclean) (unused)
ide-disk               12448   2  (autoclean)
piix                    7784   2  (autoclean)
ide-core               91832   2  (autoclean) [ide-cd ide-detect ide-disk 
piix]
unix                   12752  76  (autoclean)

The hardware doesn't seem to like the stock 2.6 kernels, and I don't have one 
spare to spend time figuring out why.

Re: Troubleshooting stability issue

[Jan Engelhardt]

>Symptom is that by the time we get to it, the box is totally unresponsive to 
>local console, is not forwarding packets. In a word "hung".

Then there may be a memory leak. Calm all apps and periodically check the free 
ram or the slab cache sizes, respectively. (Yeah, it's unlikely.)

Check /var/log/messages if it says that a network card timed out.

>Memory isn't obviously leaking.
>
>The number of lines in "ip_conntrack" does appear to grow with time, but is 
>still way below (at around 3000) the maximum allowed of 32,000+, and isn't 
>growing monotonically. I think there may be clues here, if only to what is 
>wrong with the ruleset.

Reduce the maximum number of conntracks then.

>Apart from sshd, there is practically nothing running.
>atd
>crond
>inetd (this has nothing configured in /etc/inetd.conf, so I'll remove it).
>Postfix is listening on 127.0.0.1:25 in case anything local suddenly needs to 
>report anything to me.
>lpd (lpd was running and not listening on any ports, so I'll remove it).

Where's the ftpd?

>Just looking for some helpful pointers on how to investigate this issue 
>further.

/proc/net/ip_conntrack
/proc/net/stat/ip_conntrack

>The hardware doesn't seem to like the stock 2.6 kernels, and I don't have one 
>spare to spend time figuring out why.
Forgot a driver?



Jan Engelhardt
-- 
| Alphagate Systems, http://alphagate.hopto.org/

Re: Troubleshooting stability issue

[Grant Taylor]

Simon Waters wrote:
> One of our firewalls appears to be unstable.
> 
> The problems started with adding ip_conntrack and ip_conntrack_ftp, as we need 
> to support active mode FTP from inside to the outside.
> 
> Prior to this we were using it as a basic port filter (poor iptables), and so 
> many of the rules exist to allow return packets without connection tracking.
> 
> As such it makes sense for us to "clean-up" the ruleset, but this hasn't 
> happened yet.
> 
> Due to some interesting (and maybe related?) hardware issues, we've run the 
> same firewall configuration on three different servers (all x86, Redhat and 
> Debian, IDE and SCSI), including one with the latest (for Debian) 2.6.8 
> kernel, and are fairly confident we see the same software issue on all three 
> boxes.
> 
> Symptom is that by the time we get to it, the box is totally unresponsive to 
> local console, is not forwarding packets. In a word "hung".
> 
> Memory isn't obviously leaking.

I do not recall what 2.6 kernel it was in, but there was a memory leak that plagued a system that did (extensive?) firewalling / routing in the mid 2.6 kernels.  I ran in to this on a system and ended up croning a reboot daily.  I have not had time to go back and upgrade the system to a kernel that does not have the memory leak.  I am not sure but I think I have 2.6.8 on the box.

> The number of lines in "ip_conntrack" does appear to grow with time, but is 
> still way below (at around 3000) the maximum allowed of 32,000+, and isn't 
> growing monotonically. I think there may be clues here, if only to what is 
> wrong with the ruleset.
> 
> Apart from sshd, there is practically nothing running.
> atd
> crond
> inetd (this has nothing configured in /etc/inetd.conf, so I'll remove it).
> Postfix is listening on 127.0.0.1:25 in case anything local suddenly needs to 
> report anything to me.
> lpd (lpd was running and not listening on any ports, so I'll remove it).
> 
> Just looking for some helpful pointers on how to investigate this issue 
> further.
> 
> As even with a "suboptimal" rule set I wouldn't expect the box to hang.
> Logs have no useful entries (certainly no "table full" messages).
> 
> Don't want to post the full ruleset here, at least not till I've been over it 
> with a finetoothed comb. And it is about 230 lines. On the upside quite a lot 
> of it has just been obseleted by us relocating all our machines to one site, 
> so I can shortly remove large chunks of it.
> 
>  Simon


Grant. . . .