Packet capturing performance

Packet capturing performance

[Deniz Eren]

Hi,

I'm having problem with packet capturing performance on my linux server.

I am using Intel ixgbe 10g NIC with v3.19.1 version driver over Linux
3.15.9  based system. Naturally I can route 3.8Mpps packet from spoof
(random source) addressed traffic.

Whenever I open netsniff-ng to listen interface to capture packets at
silent mode, the capturing performance slows down at the same time to
~1.2Mpps levels. I have doing pps measurements by watching the changes
at "/sys/class/net/<interface_name>/statistics/rx_packets" so the
performance can not be affected the measurements (instead of tcpstat
etc).

My first theory was bpf is cause of this slowdown. When I try to
analyze the reason of this bottleneck I see that the bpf affects the
slow down ratio. When I narrow the filter to match 1/16 packet of
traffic (for example: "src net 16.0.0.0/4" ), the capturing paket
performance stay ~3.7Mpps. And I start 16 netsniff-ng process (each
one process 1/16 part of entire traffic) with different filters the
performance stays ~3.0Mpps and the union of the 16 filter equal to
0.0.0.0/0 (0.0.0.0/4 + 16.0.0.0/4 + 32.0.0.0/4 + ...  + 248.0.0.0/4 =
0.0.0.0/0) . In other words
I think performance of network stack slow downs dramatically after a
number of matching traffic packets with given bpf.

But after some investigation and some advice from more expert people
the problem seems to be pf_packet sockets overhead. But I don't know
exactly where is the bottleneck. Do you have any idea exactly where
could be the bottleneck?

Since I am using netfilter a lot, kernel bypass is not an option for me.

To solve this problem I have two options for now:

- First one is experimenting socket fanout and adapting my tools to
use socket fanout.
- Second one is somehow similar, open more than one (ex: 16) socket
MMAP'ed socket whose have different filters from each other to match
with different part of the traffic at single netsniff_ng process. But
this one is too hacky and requires user-space modifications.

But I want to ask is there a better solution to this problem? Am I
missing a network tuning on linux or my ethernet device?

Thanks in advance,

Re: Packet capturing performance

[Eric Dumazet]

On Wed, 2015-05-20 at 16:13 +0300, Deniz Eren wrote:

> But after some investigation and some advice from more expert people
> the problem seems to be pf_packet sockets overhead. But I don't know
> exactly where is the bottleneck. Do you have any idea exactly where
> could be the bottleneck?

You use a multi queue NIC.

Make sure you setup proper IRQ affinities to serve IRQ in a sharded way.

The key is to try to have one packet queue to receive a copy of the
sniffed traffic per cpu.

AF_PACKET has PACKET_FANOUT / PACKET_FANOUT_CPU capability.

Re: Packet capturing performance

[Jesper Dangaard Brouer]

On Wed, 20 May 2015 16:13:54 +0300
Deniz Eren <denizlist@denizeren.net> wrote:

> Hi,
> 
> I'm having problem with packet capturing performance on my linux server.
> 
> I am using Intel ixgbe 10g NIC with v3.19.1 version driver over Linux
> 3.15.9  based system. Naturally I can route 3.8Mpps packet from spoof
> (random source) addressed traffic.
> 
> Whenever I open netsniff-ng to listen interface to capture packets at
> silent mode, the capturing performance slows down at the same time to
> ~1.2Mpps levels. I have doing pps measurements by watching the changes
> at "/sys/class/net/<interface_name>/statistics/rx_packets" so the
> performance can not be affected the measurements (instead of tcpstat
> etc).

Did you try to use the recently add fanout feature of netsniff-ng?

 https://github.com/netsniff-ng/netsniff-ng/commit/f00d4d54f28c0

 
> My first theory was bpf is cause of this slowdown. When I try to
> analyze the reason of this bottleneck I see that the bpf affects the
> slow down ratio. When I narrow the filter to match 1/16 packet of
> traffic (for example: "src net 16.0.0.0/4" ), the capturing paket
> performance stay ~3.7Mpps. And I start 16 netsniff-ng process (each
> one process 1/16 part of entire traffic) with different filters the
> performance stays ~3.0Mpps and the union of the 16 filter equal to
> 0.0.0.0/0 (0.0.0.0/4 + 16.0.0.0/4 + 32.0.0.0/4 + ...  + 248.0.0.0/4 =
> 0.0.0.0/0) . In other words
> I think performance of network stack slow downs dramatically after a
> number of matching traffic packets with given bpf.
> 
> But after some investigation and some advice from more expert people
> the problem seems to be pf_packet sockets overhead. But I don't know
> exactly where is the bottleneck. Do you have any idea exactly where
> could be the bottleneck?
> 
> Since I am using netfilter a lot, kernel bypass is not an option for me.
> 
> To solve this problem I have two options for now:
> 
> - First one is experimenting socket fanout and adapting my tools to
> use socket fanout.
> - Second one is somehow similar, open more than one (ex: 16) socket
> MMAP'ed socket whose have different filters from each other to match
> with different part of the traffic at single netsniff_ng process. But
> this one is too hacky and requires user-space modifications.
> 
> But I want to ask is there a better solution to this problem? Am I
> missing a network tuning on linux or my ethernet device?


-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Sr. Network Kernel Developer at Red Hat
  Author of http://www.iptv-analyzer.org
  LinkedIn: http://www.linkedin.com/in/brouer

Re: Packet capturing performance

[yzhu1]

Hi,

You can use netfilter to mirror the packets to another nic. Then you can
capture these cloned patckets on this nic.

It can not affect the performance. Believe me, I made tests with it.

Zhu Yanjun

On 05/20/2015 09:13 PM, Deniz Eren wrote:
> Hi,
>
> I'm having problem with packet capturing performance on my linux server.
>
> I am using Intel ixgbe 10g NIC with v3.19.1 version driver over Linux
> 3.15.9  based system. Naturally I can route 3.8Mpps packet from spoof
> (random source) addressed traffic.
>
> Whenever I open netsniff-ng to listen interface to capture packets at
> silent mode, the capturing performance slows down at the same time to
> ~1.2Mpps levels. I have doing pps measurements by watching the changes
> at "/sys/class/net/<interface_name>/statistics/rx_packets" so the
> performance can not be affected the measurements (instead of tcpstat
> etc).
>
> My first theory was bpf is cause of this slowdown. When I try to
> analyze the reason of this bottleneck I see that the bpf affects the
> slow down ratio. When I narrow the filter to match 1/16 packet of
> traffic (for example: "src net 16.0.0.0/4" ), the capturing paket
> performance stay ~3.7Mpps. And I start 16 netsniff-ng process (each
> one process 1/16 part of entire traffic) with different filters the
> performance stays ~3.0Mpps and the union of the 16 filter equal to
> 0.0.0.0/0 (0.0.0.0/4 + 16.0.0.0/4 + 32.0.0.0/4 + ...  + 248.0.0.0/4 =
> 0.0.0.0/0) . In other words
> I think performance of network stack slow downs dramatically after a
> number of matching traffic packets with given bpf.
>
> But after some investigation and some advice from more expert people
> the problem seems to be pf_packet sockets overhead. But I don't know
> exactly where is the bottleneck. Do you have any idea exactly where
> could be the bottleneck?
>
> Since I am using netfilter a lot, kernel bypass is not an option for me.
>
> To solve this problem I have two options for now:
>
> - First one is experimenting socket fanout and adapting my tools to
> use socket fanout.
> - Second one is somehow similar, open more than one (ex: 16) socket
> MMAP'ed socket whose have different filters from each other to match
> with different part of the traffic at single netsniff_ng process. But
> this one is too hacky and requires user-space modifications.
>
> But I want to ask is there a better solution to this problem? Am I
> missing a network tuning on linux or my ethernet device?
>
> Thanks in advance,
> --
> To unsubscribe from this list: send the line "unsubscribe netdev" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>