Re: Strange packet drops with heavy firewalling

[Eric Dumazet <eric.dumazet@gmail.com>]

Le vendredi 09 avril 2010 à 11:56 +0200, Benny Amorsen a écrit :
> I have a netfilter-box which is dropping packets. ethtool -S counts
> 10-20 rx_discards per second on the interface.
> 
> The switch does not have flow control enabled; with flow control enabled
> the rx_discards turn into tx_on_sent which ultimately cause the same
> problem (the load is pretty constant so the switch has to drop the
> packets instead).
> 
> perf top shows something like:
>              5201.00 -  6.7% : _spin_unlock_irqrestore
>              4232.00 -  5.5% : finish_task_switch
>              3597.00 -  4.6% : tg3_poll	[tg3]
>              3257.00 -  4.2% : handle_IRQ_event
>              2515.00 -  3.2% : tick_nohz_restart_sched_tick
>              1947.00 -  2.5% : nf_ct_tuple_equal
>              1927.00 -  2.5% : tg3_start_xmit	[tg3]
>              1879.00 -  2.4% : kmem_cache_alloc_node
>              1625.00 -  2.1% : tick_nohz_stop_sched_tick
>              1619.00 -  2.1% : ipt_do_table
>              1595.00 -  2.1% : ip_route_input
>              1547.00 -  2.0% : kmem_cache_free
>              1474.00 -  1.9% : __alloc_skb
>              1424.00 -  1.8% : fget_light
>              1391.00 -  1.8% : nf_iterate
> 
> The rule set is quite large (more than 4000 rules), but organized so
> that each packet only has to traverse a few rules before getting
> accepted or rejected.
> 
> When the problem started we were using a different server, an old
> two-socket 32-bit Xeon with hyperthreading. CPU usage often hit 100% on
> one CPU with that server. After replacing the server with a ProLiant
> DL160 G5 with a quad-core Xeon (without hyperthreading) the CPU usage
> rarely exceeds 10% on any CPU, but the packet loss persists.
> 

might be micro bursts, check 'ethtool -g eth0' RX parameters (increase
RX ring from 200 to 511 if you want more buffers ?)

> We're using the built-in dual Broadcom Corporation NetXtreme BCM5722 Gigabit
> Ethernet PCI Express nics, and the kernel is
> kernel-2.6.32.9-70.fc12.x86_64 from Fedora. Next step is probably
> installing a better ethernet card, perhaps an Intel 82576-based one, so
> that we can get multiqueue support.
> 

Sure, but before this, could you check

cat /proc/net/softnet_stat
cat /proc/interrupts
(check eth0 IRQS are delivered to one cpu)

grep . /proc/sys/net/ipv4/netfilter/ip_conntrack_*
(might need to increase ip_conntrack_buckets)

ethtool -c eth0
(might change coalesce params to reduce number of irqs)

ethtool -g eth0