RE: Troubleshooting Netfilter Firewall (performance issues)

RE: Troubleshooting Netfilter Firewall (performance issues)

[Derick Anderson]

Inline... 

> -----Original Message-----
> From: netfilter-bounces@lists.netfilter.org 
> [mailto:netfilter-bounces@lists.netfilter.org] On Behalf Of 
> Harrison, James
> Sent: Thursday, November 03, 2005 12:28 PM
> To: netfilter@lists.netfilter.org
> Subject: Troubleshooting Netfilter Firewall (performance issues)
> 
> List,
> 
> I am currently troubleshooting performance issues on a 
> network that seem to indicate an issue with the firewall.
> 
> I've been using a netfilter configuration for almost 2 years 
> without issue, but we have been suffering through lost 
> connections(tcp resets) when transferring files through the 
> firewall via scp, ftp, http, and smb.
> 
> All interfaces on the firewall and on the switches connected 
> to the firewall appear clean.
> 
> Can someone help me troubleshoot this to determine what might 
> be going on?  I've used fwbuilder to build the ruleset and up 
> until ~10/7/2005 we were not experiencing any issues whatsoever.

If you can post the output of iptables-save that might help. What kind
of NICs are you using? Are they Gigabit? What kernel? What distro?
(assuming RedHat from the sig) Have you done anything to the kernel
settings? What have you done so far to troubleshoot the problem?

> The current box is a dual processor(1400Mhz) Dell 1650:
> 
> top says:
> top - 11:26:42 up 9 days, 10:21,  3 users,  load average: 
> 0.05, 0.06, 0.06
> Tasks:  51 total,   1 running,  50 sleeping,   0 stopped,   0 zombie
> Cpu(s):   0.0% user,   1.0% system,   0.0% nice,  99.0% idle
> Mem:   1032992k total,   124124k used,   908868k free,     
> 9744k buffers
> Swap:        0k total,        0k used,        0k free,    
> 28660k cached

So memory isn't the issue, and I wouldn't think CPU... The firewall I
administer runs with a quarter of that RAM on a fairly busy 3mbit
(duplex) line, although it has a faster CPU.

> netstat -s says:
> 
> Ip:
>     803083052 total packets received
>     797709786 forwarded
>     0 incoming packets discarded
>     1206740 incoming packets delivered
>     2326772 requests sent out
>     1008 outgoing packets dropped
>     11 fragments dropped after timeout
>     222078 reassemblies required
>     11467 packets reassembled ok
>     11 packet reassembles failed
>     11184 fragments received ok
>     221228 fragments created
> Icmp:
>     45 ICMP messages received
>     10 input ICMP message failed.
>     ICMP input histogram:
>         destination unreachable: 2
>         echo requests: 33
>     13281 ICMP messages sent
>     0 ICMP messages failed
>     ICMP output histogram:
>         destination unreachable: 1610
>         time exceeded: 11671
> Tcp:
>     4 active connections openings
>     114 passive connection openings
>     2 failed connection attempts
>     12 connection resets received
>     3 connections established
>     382150 segments received
>     834405 segments send out
>     10239 segments retransmited
>     0 bad segments received.
>     330 resets sent
> Udp:
>     819564 packets received
>     319 packets to unknown port received.
>     0 packet receive errors
>     1479565 packets sent
> TcpExt:
>     58 invalid SYN cookies received
>     29 TCP sockets finished time wait in fast timer
>     7 packets rejects in established connections because of timestamp
>     576 delayed acks sent
>     27 delayed acks further delayed because of locked socket
>     Quick ack mode was activated 58 times
>     181 packets directly queued to recvmsg prequeue.
>     193 of bytes directly received from prequeue
>     10144 packet headers predicted
>     4 packets header predicted and directly queued to user
>     56472 acknowledgments not containing data received
>     302400 predicted acknowledgments
>     170 times recovered from packet loss due to SACK data
>     Detected reordering 6 times using time stamp
>     4 congestion windows fully recovered
>     82 congestion windows partially recovered using Hoe heuristic
>     TCPDSACKUndo: 1
>     1035 congestion windows recovered after partial ack
>     88 TCP data loss events
>     27 timeouts after SACK recovery
>     374 fast retransmits
>     29 forward retransmits
>     8106 retransmits in slow start
>     1610 other TCP timeouts
>     5 sack retransmits failed
>     58 DSACKs sent for old packets
>     3003 DSACKs received
>     5 DSACKs for out of order packets received
>     6 connections aborted due to timeout

I looked at my output for netstat -s and see similar outputs except when
it comes to fragments and resets...

> Thanks
> --
> James Harrison RHCE
> Manager, Information Security
> AIM: harrijh1

If I were you I would monitor top during a large transfer and maybe do
an ethereal dump as well. If your two endpoint machines are both on Gbit
LAN and your firewall is 100Mbit (on a 100/1000 switch) then perhaps
your firewall NICs are getting overloaded. Every night at my company all
the servers (Gbit) back up to a local machine (100Mbit). They each have
their time window for backing up but it's common for Nagios to report an
"UNKNOWN" status for the backup server in the early morning hours. Of
course that could simply be the poor little backup server not having the
time to reply...

Derick Anderson

RE: Troubleshooting Netfilter Firewall (performance issues)

[Harrison, James]

On Thu, 2005-11-03 at 12:55 -0500, Derick Anderson wrote:
[snip]

> 
> If I were you I would monitor top during a large transfer and maybe do
> an ethereal dump as well. If your two endpoint machines are both on Gbit
> LAN and your firewall is 100Mbit (on a 100/1000 switch) then perhaps
> your firewall NICs are getting overloaded. Every night at my company all
> the servers (Gbit) back up to a local machine (100Mbit). They each have
> their time window for backing up but it's common for Nagios to report an
> "UNKNOWN" status for the backup server in the early morning hours. Of
> course that could simply be the poor little backup server not having the
> time to reply...
> 
> Derick Anderson

According to netstat -i I shouldn't be having issues with overloading
the interfaces. (TX-ERR on eth0 and eth1 are static and have not
incremented) I have 2/100MB and 2/1000MB interfaces.

Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR TX-DRP TX-OVR Flg
eth0   1500 0  50757220      0      0      050703586    523      0      0 BMRU
eth0:  1500 0       - no statistics available -                        BMRU
eth1   1500 0  339989009      0      0      0397086634   3381      0      0 BMRU
eth1:  1500 0       - no statistics available -                        BMRU
eth2   1500 0  409181861      0      0      0344550753      0      0      0 BMRU
eth3   1500 0  11352902      0      0      015003672      0      0      0 BMRU
eth3:  1500 0       - no statistics available -                        BMRU

It is running on the Devil Linux distro.

-- 
James Harrison RHCE
Manager, Information Security
AIM: harrijh1

RE: Troubleshooting Netfilter Firewall (performance issues)

[Derick Anderson]

 

> -----Original Message-----
> From: Harrison, James [mailto:james.harrison@americancolor.com] 
> Sent: Thursday, November 03, 2005 1:24 PM
> To: Derick Anderson
> Cc: netfilter@lists.netfilter.org
> Subject: RE: Troubleshooting Netfilter Firewall (performance issues)
> 
> On Thu, 2005-11-03 at 12:55 -0500, Derick Anderson wrote:
> [snip]
> 
> > 
> > If I were you I would monitor top during a large transfer 
> and maybe do 
> > an ethereal dump as well. If your two endpoint machines are both on 
> > Gbit LAN and your firewall is 100Mbit (on a 100/1000 switch) then 
> > perhaps your firewall NICs are getting overloaded. Every 
> night at my 
> > company all the servers (Gbit) back up to a local machine 
> (100Mbit). 
> > They each have their time window for backing up but it's common for 
> > Nagios to report an "UNKNOWN" status for the backup server in the 
> > early morning hours. Of course that could simply be the poor little 
> > backup server not having the time to reply...
> > 
> > Derick Anderson
> 
> According to netstat -i I shouldn't be having issues with 
> overloading the interfaces. (TX-ERR on eth0 and eth1 are 
> static and have not
> incremented) I have 2/100MB and 2/1000MB interfaces.
> 
> Iface   MTU Met   RX-OK RX-ERR RX-DRP RX-OVR   TX-OK TX-ERR 
> TX-DRP TX-OVR Flg
> eth0   1500 0  50757220      0      0      050703586    523   
>    0      0 BMRU
> eth0:  1500 0       - no statistics available -               
>          BMRU
> eth1   1500 0  339989009      0      0      0397086634   3381 
>      0      0 BMRU
> eth1:  1500 0       - no statistics available -               
>          BMRU
> eth2   1500 0  409181861      0      0      0344550753      0 
>      0      0 BMRU
> eth3   1500 0  11352902      0      0      015003672      0   
>    0      0 BMRU
> eth3:  1500 0       - no statistics available -               
>          BMRU
> 
> It is running on the Devil Linux distro.
> 
> --
> James Harrison RHCE
> Manager, Information Security
> AIM: harrijh1

Unless Devil Linux has messed with your TCP/IP options in the kernel,
then I think it's time to look at the firewall. If you have an
abnormally high number of rules, this could be an issue, or if you're
rate-limiting anything...

Derick Anderson

Troubleshooting Netfilter Firewall (performance issues)

[Harrison, James]

List,

I am currently troubleshooting performance issues on a network that seem
to indicate an issue with the firewall.

I've been using a netfilter configuration for almost 2 years without
issue, but we have been suffering through lost connections(tcp resets)
when transferring files through the firewall via scp, ftp, http, and
smb.

All interfaces on the firewall and on the switches connected to the
firewall appear clean.

Can someone help me troubleshoot this to determine what might be going
on?  I've used fwbuilder to build the ruleset and up until ~10/7/2005 we
were not experiencing any issues whatsoever.

The current box is a dual processor(1400Mhz) Dell 1650:

top says:
top - 11:26:42 up 9 days, 10:21,  3 users,  load average: 0.05, 0.06, 0.06
Tasks:  51 total,   1 running,  50 sleeping,   0 stopped,   0 zombie
Cpu(s):   0.0% user,   1.0% system,   0.0% nice,  99.0% idle
Mem:   1032992k total,   124124k used,   908868k free,     9744k buffers
Swap:        0k total,        0k used,        0k free,    28660k cached



netstat -s says:

Ip:
    803083052 total packets received
    797709786 forwarded
    0 incoming packets discarded
    1206740 incoming packets delivered
    2326772 requests sent out
    1008 outgoing packets dropped
    11 fragments dropped after timeout
    222078 reassemblies required
    11467 packets reassembled ok
    11 packet reassembles failed
    11184 fragments received ok
    221228 fragments created
Icmp:
    45 ICMP messages received
    10 input ICMP message failed.
    ICMP input histogram:
        destination unreachable: 2
        echo requests: 33
    13281 ICMP messages sent
    0 ICMP messages failed
    ICMP output histogram:
        destination unreachable: 1610
        time exceeded: 11671
Tcp:
    4 active connections openings
    114 passive connection openings
    2 failed connection attempts
    12 connection resets received
    3 connections established
    382150 segments received
    834405 segments send out
    10239 segments retransmited
    0 bad segments received.
    330 resets sent
Udp:
    819564 packets received
    319 packets to unknown port received.
    0 packet receive errors
    1479565 packets sent
TcpExt:
    58 invalid SYN cookies received
    29 TCP sockets finished time wait in fast timer
    7 packets rejects in established connections because of timestamp
    576 delayed acks sent
    27 delayed acks further delayed because of locked socket
    Quick ack mode was activated 58 times
    181 packets directly queued to recvmsg prequeue.
    193 of bytes directly received from prequeue
    10144 packet headers predicted
    4 packets header predicted and directly queued to user
    56472 acknowledgments not containing data received
    302400 predicted acknowledgments
    170 times recovered from packet loss due to SACK data
    Detected reordering 6 times using time stamp
    4 congestion windows fully recovered
    82 congestion windows partially recovered using Hoe heuristic
    TCPDSACKUndo: 1
    1035 congestion windows recovered after partial ack
    88 TCP data loss events
    27 timeouts after SACK recovery
    374 fast retransmits
    29 forward retransmits
    8106 retransmits in slow start
    1610 other TCP timeouts
    5 sack retransmits failed
    58 DSACKs sent for old packets
    3003 DSACKs received
    5 DSACKs for out of order packets received
    6 connections aborted due to timeout


Thanks
-- 
James Harrison RHCE
Manager, Information Security
AIM: harrijh1