From mboxrd@z Thu Jan  1 00:00:00 1970
From: Aaron Lu <aaron.lu@intel.com>
Subject: Re: Kernel 4.19 network performance - forwarding/routing normal
 users traffic
Date: Fri, 2 Nov 2018 13:23:56 +0800
Message-ID: <20181102052356.GA17587@intel.com>
References: <61697e49-e839-befc-8330-fc00187c48ee@itcare.pl>
 <61e30474-b5e9-4dc8-a8a6-90cdd17d2a66@gmail.com>
 <8e10bf68-f3b3-98f2-91a5-25b151756dd6@itcare.pl>
 <20181101102213.2fa2643d@redhat.com>
 <20181101152716.GA13895@intel.com>
 <a7cfa596d6b9979c67fa8dbd633f7dd8293337a1.camel@mellanox.com>
Mime-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Cc: "brouer@redhat.com" <brouer@redhat.com>,
        "pstaszewski@itcare.pl" <pstaszewski@itcare.pl>,
        "eric.dumazet@gmail.com" <eric.dumazet@gmail.com>,
        "netdev@vger.kernel.org" <netdev@vger.kernel.org>,
        Tariq Toukan <tariqt@mellanox.com>,
        "ilias.apalodimas@linaro.org" <ilias.apalodimas@linaro.org>,
        "yoel@kviknet.dk" <yoel@kviknet.dk>,
        "mgorman@techsingularity.net" <mgorman@techsingularity.net>
To: Saeed Mahameed <saeedm@mellanox.com>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mga09.intel.com ([134.134.136.24]:18765 "EHLO mga09.intel.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1727875AbeKBO3x (ORCPT <rfc822;netdev@vger.kernel.org>);
        Fri, 2 Nov 2018 10:29:53 -0400
Content-Disposition: inline
In-Reply-To: <a7cfa596d6b9979c67fa8dbd633f7dd8293337a1.camel@mellanox.com>
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

On Thu, Nov 01, 2018 at 08:23:19PM +0000, Saeed Mahameed wrote:
> On Thu, 2018-11-01 at 23:27 +0800, Aaron Lu wrote:
> > On Thu, Nov 01, 2018 at 10:22:13AM +0100, Jesper Dangaard Brouer
> > wrote:
> > ... ...
> > > Section copied out:
> > > 
> > >   mlx5e_poll_tx_cq
> > >   |          
> > >    --16.34%--napi_consume_skb
> > >              |          
> > >              |--12.65%--__free_pages_ok
> > >              |          |          
> > >              |           --11.86%--free_one_page
> > >              |                     |          
> > >              |                     |--10.10%
> > > --queued_spin_lock_slowpath
> > >              |                     |          
> > >              |                      --0.65%--_raw_spin_lock
> > 
> > This callchain looks like it is freeing higher order pages than order
> > 0:
> > __free_pages_ok is only called for pages whose order are bigger than
> > 0.
> 
> mlx5 rx uses only order 0 pages, so i don't know where these high order
> tx SKBs are coming from.. 

Perhaps here:
__netdev_alloc_skb(), __napi_alloc_skb(), __netdev_alloc_frag() and
__napi_alloc_frag() will all call page_frag_alloc(), which will use
__page_frag_cache_refill() to get an order 3 page if possible, or fall
back to an order 0 page if order 3 page is not available.

I'm not sure if your workload will use the above code path though.