Regarding bottlenecks for high speed packet generation

Regarding bottlenecks for high speed packet generation

[Ajith Adapa]

Hi,

I am trying out a sample application written based on packet_mmap to
generate packets at line rates. I am using 3.4.10 kernel and a gigabit
nic card.
I have found some strange issues as mentioned below.

When I am transmitting packets sized 1500 bytes the socket buffers are
easily filled up eventhough if I increase the value of wmem_default
and wmem_max.
The memory I have allotted can fit around 14k packets of 1500 bytes
size. Is the NIC card not able to transmit them ?? How can I check it
??
I have even increased the Hardware tx queue size of NIC card from 256
to 4096 using ethtool.

Does traffic control causing any issue as I read it wont allow traffic bursts ?

Is there any way to find the major bottlenecks that would really cause
problems in high-speed packet generation.

Regards,
Ajith

Re: Regarding bottlenecks for high speed packet generation

[Daniel Borkmann]

On Thu, Oct 25, 2012 at 7:46 PM, Ajith Adapa <adapa.ajith@gmail.com> wrote:
> I am trying out a sample application written based on packet_mmap to
> generate packets at line rates. I am using 3.4.10 kernel and a gigabit
> nic card.
> I have found some strange issues as mentioned below.
>
> When I am transmitting packets sized 1500 bytes the socket buffers are
> easily filled up eventhough if I increase the value of wmem_default
> and wmem_max.
> The memory I have allotted can fit around 14k packets of 1500 bytes
> size. Is the NIC card not able to transmit them ?? How can I check it
> ??
> I have even increased the Hardware tx queue size of NIC card from 256
> to 4096 using ethtool.
>
> Does traffic control causing any issue as I read it wont allow traffic bursts ?

Actually in my experience not. On Gigabit Ethernet (depending on your
machine) you should be able to generate about 80k pps with 1500 byte
(e.g. with netsniff-ng's trafgen) having the default tc discipline.

> Is there any way to find the major bottlenecks that would really cause
> problems in high-speed packet generation.

Have you looked how much pps the kernel space pktgen can generate on
your machine with 1500 byte packets?

What tool do you use to measure this reported rate? Do you analyze
driver statistics from e.g. procfs, or do you use a libpcap-based tool
to measure the stuff (which you shouldn't since it could falsify your
stats)? How was your measurement setup?

Re: Regarding bottlenecks for high speed packet generation

[Daniel Borkmann]

(Please reply to all, so that also netdev gets CC'ed as well.)

On Thu, Oct 25, 2012 at 8:25 PM, Ajith Adapa <adapa.ajith@gmail.com> wrote:
>> Actually in my experience not. On Gigabit Ethernet (depending on your
>> machine) you should be able to generate about 80k pps with 1500 byte
>> (e.g. with netsniff-ng's trafgen) having the default tc discipline.
>>
> Currently I can see that my machine is able to generate 80MBytes/sec
> for 1500 byte packets with
> default tc discipline. Its a simple desktop with i5 dual core machine, 4GB RAM.

Hmm, what is the interconnect of the NIC? PCIe? I've seen some _PCI_
devices on simple desktop machines where this number could fit.

>> Have you looked how much pps the kernel space pktgen can generate on
>> your machine with 1500 byte packets?
>>
>> What tool do you use to measure this reported rate? Do you analyze
>> driver statistics from e.g. procfs, or do you use a libpcap-based tool
>> to measure the stuff (which you shouldn't since it could falsify your
>> stats)? How was your measurement setup?
>
> Currently I am using IXIA for the destination to confirm the numbers
> and also basic stats from system monitor.

Ok.

> I have tried pktgen quite long back but I could make a try and confirm
> the rate at which generates on my machine.

Yes, might be worth a try.

> When I tried powertop tool I see that out of list processes who
> consume more amount of power I can see that the NIC interface
> on which I am trying to burst the traffic is consuming around 63% with
> interrupts. But does the NIC card drop a interrupt for
> every packet which it has successfully sent ?

If the driver uses NAPI [1] / the chip supports that, then not. You
can observe the hardware / software interrupts via ifpps or watching
procfs or some other tools, or simply look at the driver
implementation.

[1] http://www.linuxfoundation.org/collaborate/workgroups/networking/napi

Re: Regarding bottlenecks for high speed packet generation

[Ajith Adapa]

Hi,

> If the driver uses NAPI [1] / the chip supports that, then not. You
> can observe the hardware / software interrupts via ifpps or watching
> procfs or some other tools, or simply look at the driver
> implementation.
>
> [1] http://www.linuxfoundation.org/collaborate/workgroups/networking/napi

But does NAPI comes into picture in case of transmission of packets ?
I read it comes into picture
only in the case of reception of packets.

Regards,
Ajith

Re: Regarding bottlenecks for high speed packet generation

[David Miller]

From: Ajith Adapa <adapa.ajith@gmail.com>
Date: Fri, 26 Oct 2012 08:19:15 +0530

> Hi,
> 
>> If the driver uses NAPI [1] / the chip supports that, then not. You
>> can observe the hardware / software interrupts via ifpps or watching
>> procfs or some other tools, or simply look at the driver
>> implementation.
>>
>> [1] http://www.linuxfoundation.org/collaborate/workgroups/networking/napi
> 
> But does NAPI comes into picture in case of transmission of packets ?

Reclaim of TX descriptors occurs via NAPI.

Re: Regarding bottlenecks for high speed packet generation

[Ajith Adapa]

> >
> > But does NAPI comes into picture in case of transmission of packets ?
>
> Reclaim of TX descriptors occurs via NAPI.

You mean for every packet sent by the NIC card it drops a interrupt to
reclaim the TX buffers used ?
I am not able to understand why does NIC has to interrupt back instead
of freeing the memory in background.

Might be it does add some bottleneck in my case of transmitting traffic bursts.

Regards,
Ajith

Re: Regarding bottlenecks for high speed packet generation

[Ben Hutchings]

On Fri, 2012-10-26 at 11:42 +0530, Ajith Adapa wrote:
> > >
> > > But does NAPI comes into picture in case of transmission of packets ?
> >
> > Reclaim of TX descriptors occurs via NAPI.
> 
> You mean for every packet sent by the NIC card it drops a interrupt to
> reclaim the TX buffers used ?

Not necessarily; completions may be batched in some way.

> I am not able to understand why does NIC has to interrupt back instead
> of freeing the memory in background.
>
> Might be it does add some bottleneck in my case of transmitting traffic bursts.

The network core needs to account and limit how much memory is allocated
to socket buffers.  Completed TX buffers should therefore be freed
promptly, to avoid blocking the sending socket unnecessarily.  It
doesn't strictly matter whether this is triggered by an IRQ from the NIC
or by a timer, but it certainly must not be triggered by the next TX
initiation as this may result in a kind of deadlock.

Ben.

-- 
Ben Hutchings, Staff Engineer, Solarflare
Not speaking for my employer; that's the marketing department's job.
They asked us to note that Solarflare product names are trademarked.

Re: Regarding bottlenecks for high speed packet generation

[Ajith Adapa]

Hi,

> The network core needs to account and limit how much memory is allocated
> to socket buffers.  Completed TX buffers should therefore be freed
> promptly, to avoid blocking the sending socket unnecessarily.  It
> doesn't strictly matter whether this is triggered by an IRQ from the NIC
> or by a timer, but it certainly must not be triggered by the next TX
> initiation as this may result in a kind of deadlock.

My understanding is when an SKB is created we create it from the
amount of socket send buffer and sent to
device driver which sends to the hardware for transmission. Once the
NIC is done transmitting the packet an
interrupt allows the SKB to be freed returning the same amount of
memory back to socket send buffer.
I can see that even if I allocate 150MB for socket TX buffer size
after generating around 14k packets
the memory is exhausted. So ideally it means NIC is not able to
transmit the data at the rate it is being
received data from user.

As @dan mentioned earlier I have tried pktgen generating a 1500byte
packets. I can reach not more than
72MB/s but the CPU utilization has peaked 80% on the core on which
pktgen is binded with.

Regards,
Ajith