netif_receive_skb is taking long time

netif_receive_skb is taking long time

[Keyur Amrutbhai Patel]

Hi,

In my NIC driver "netif_receive_skb" is taking too long time. Almost 3375 neno seconds. Which is more than whole packet processing from interrupt.

Could anyone please help me to understand what could be the reason behind this? How to solve it to take minimum time?

Is there any standard calls which we need to follow in order to get faster performance?

Regards,
Keyur


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

RE: netif_receive_skb is taking long time

[Keyur Amrutbhai Patel]

Any help on this would be appreciated. 

-----Original Message-----
From: netdev-owner@vger.kernel.org <netdev-owner@vger.kernel.org> On Behalf Of Keyur Amrutbhai Patel
Sent: Thursday, October 25, 2018 9:09 PM
To: netdev@vger.kernel.org
Subject: netif_receive_skb is taking long time

Hi,

In my NIC driver "netif_receive_skb" is taking too long time. Almost 3375 neno seconds. Which is more than whole packet processing from interrupt.

Could anyone please help me to understand what could be the reason behind this? How to solve it to take minimum time?

Is there any standard calls which we need to follow in order to get faster performance?

Regards,
Keyur


This email and any attachments are intended for the sole use of the named recipient(s) and contain(s) confidential information that may be proprietary, privileged or copyrighted under applicable law. If you are not the intended recipient, do not read, copy, or forward this email message or any attachments. Delete this email message and any attachments immediately.

Re: netif_receive_skb is taking long time

[Eric Dumazet]

On 10/25/2018 08:39 AM, Keyur Amrutbhai Patel wrote:
> Hi,
> 
> In my NIC driver "netif_receive_skb" is taking too long time. Almost 3375 neno seconds. Which is more than whole packet processing from interrupt.
> 
> Could anyone please help me to understand what could be the reason behind this? How to solve it to take minimum time?
> 
> Is there any standard calls which we need to follow in order to get faster performance?
> 

First step would be to read Documentation/networking/scaling.txt and see if anything there helps.

Have you tried to profile the kernel and see if some contention or hot function appears ?

Maybe use a faster cpu, or remove not needed features like too heavy netfilter rules.

We can not really answer your question, you do not provide enough information.

RE: netif_receive_skb is taking long time

[Keyur Amrutbhai Patel]

Hi Eric,

First of all thank you for replying and giving some spotlight.

First step would be to read Documentation/networking/scaling.txt and see if anything there helps.
 - This is good article. I had gone through it.  Any suggestion on RSS? How to configure it? Do I need to take care anything specially in my NIC driver?

Have you tried to profile the kernel and see if some contention or hot function appears ?
- I have added time stampings in different functions. That is how I came to know that almost ~3375 neno seconds are used by just " netif_receive_skb " don’t know why. With less than that time my DMA operation is finishes and descriptors are managed.
Current time consuming function are " netif_receive_skb " and " napi_alloc_skb " these two function calls are taking maximum about of time

Maybe use a faster cpu, or remove not needed features like too heavy netfilter rules.
- I am using Intex Xeon Platinum series processors. These are fast enough CPUs available in market with 64 cores. 2 CPU nodes (each has 32 core)

We can not really answer your question, you do not provide enough information.
- Please let me know what additional details you need. We have 6 queues in HW. Each is mapped to MSI-X vector. Each vector is giving interrupt on different CPU. From interrupt I am scheduling napi and from napi poll function I am getting DMA page and constructing skb and passing it to network layer with "netif_receive_skb".

Let me know additional details which are required.

Regards,
Keyur

-----Original Message-----
From: Eric Dumazet <eric.dumazet@gmail.com> 
Sent: Thursday, October 25, 2018 10:38 PM
To: Keyur Amrutbhai Patel <keyurp@xilinx.com>; netdev@vger.kernel.org
Subject: Re: netif_receive_skb is taking long time

EXTERNAL EMAIL

On 10/25/2018 08:39 AM, Keyur Amrutbhai Patel wrote:
> Hi,
>
> In my NIC driver "netif_receive_skb" is taking too long time. Almost 3375 neno seconds. Which is more than whole packet processing from interrupt.
>
> Could anyone please help me to understand what could be the reason behind this? How to solve it to take minimum time?
>
> Is there any standard calls which we need to follow in order to get faster performance?
>

First step would be to read Documentation/networking/scaling.txt and see if anything there helps.

Have you tried to profile the kernel and see if some contention or hot function appears ?

Maybe use a faster cpu, or remove not needed features like too heavy netfilter rules.

We can not really answer your question, you do not provide enough information.

Re: netif_receive_skb is taking long time

[Eric Dumazet]

Please do not top post, and use normal quoting.

On 10/25/2018 10:22 AM, Keyur Amrutbhai Patel wrote:
> Hi Eric,
> 
> First of all thank you for replying and giving some spotlight.
> 
> First step would be to read Documentation/networking/scaling.txt and see if anything there helps.
>  - This is good article. I had gone through it.  Any suggestion on RSS? How to configure it? Do I need to take care anything specially in my NIC driver?

Just read the page and apply the various configurations.

> 
> Have you tried to profile the kernel and see if some contention or hot function appears ?
> - I have added time stampings in different functions. That is how I came to know that almost ~3375 neno seconds are used by just " netif_receive_skb " don’t know why. With less than that time my DMA operation is finishes and descriptors are managed.
> Current time consuming function are " netif_receive_skb " and " napi_alloc_skb " these two function calls are taking maximum about of time
> 

So... networking spend more time in upper stacks than a driver.

A driver does almost nothing, just passing around bits that that NIC put in memory.

In most workloads, a driver would not use more than 5% of total cpu cycles.

Now, if all you need is to impress your friends/boss about some
crazy number of RX packets per second,
just do not allocate skbs, and not call netif_receive_skb(),
use something like XDP to drop incoming frames :)

> Maybe use a faster cpu, or remove not needed features like too heavy netfilter rules.
> - I am using Intex Xeon Platinum series processors. These are fast enough CPUs available in market with 64 cores. 2 CPU nodes (each has 32 core)
> 
> We can not really answer your question, you do not provide enough information.
> - Please let me know what additional details you need. We have 6 queues in HW. Each is mapped to MSI-X vector. Each vector is giving interrupt on different CPU. From interrupt I am scheduling napi and from napi poll function I am getting DMA page and constructing skb and passing it to network layer with "netif_receive_skb".
> 
> Let me know additional details which are required.
> 
> Regards,
> Keyur
> 
> -----Original Message-----
> From: Eric Dumazet <eric.dumazet@gmail.com> 
> Sent: Thursday, October 25, 2018 10:38 PM
> To: Keyur Amrutbhai Patel <keyurp@xilinx.com>; netdev@vger.kernel.org
> Subject: Re: netif_receive_skb is taking long time
> 
> EXTERNAL EMAIL
> 
> On 10/25/2018 08:39 AM, Keyur Amrutbhai Patel wrote:
>> Hi,
>>
>> In my NIC driver "netif_receive_skb" is taking too long time. Almost 3375 neno seconds. Which is more than whole packet processing from interrupt.
>>
>> Could anyone please help me to understand what could be the reason behind this? How to solve it to take minimum time?
>>
>> Is there any standard calls which we need to follow in order to get faster performance?
>>
> 
> First step would be to read Documentation/networking/scaling.txt and see if anything there helps.
> 
> Have you tried to profile the kernel and see if some contention or hot function appears ?
> 
> Maybe use a faster cpu, or remove not needed features like too heavy netfilter rules.
> 
> We can not really answer your question, you do not provide enough information.
> 

Re: netif_receive_skb is taking long time

[David Miller]

From: Keyur Amrutbhai Patel <keyurp@xilinx.com>
Date: Thu, 25 Oct 2018 17:22:02 +0000

> Current time consuming function are " netif_receive_skb " and "
> napi_alloc_skb " these two function calls are taking maximum about
> of time

netif_receive_skb() calls the entire networking stack receive path.
So measuring it by itself it not very useful.

Use 'perf' or a similar tool to fully profile the kernel and get a
more detailed analysis.