From mboxrd@z Thu Jan  1 00:00:00 1970
From: Jesper Dangaard Brouer <brouer@redhat.com>
Subject: Re: [net-next, RFC, 4/8] net: core: add recycle capabilities on
 skbs via page_pool API
Date: Sat, 8 Dec 2018 12:36:10 +0100
Message-ID: <20181208123610.7990e85d@redhat.com>
References: <154413874729.21735.10644578158550468689.stgit@firesoul>
        <20181208095758.GA32028@strlen.de>
Mime-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
Cc: netdev@vger.kernel.org, "David S. Miller" <davem@davemloft.net>,
        Toke =?UTF-8?B?SMO4aWxhbmQtSsO4cmdlbnNlbg==?= <toke@toke.dk>,
        ard.biesheuvel@linaro.org, Jason Wang <jasowang@redhat.com>,
        ilias.apalodimas@linaro.org,
        =?UTF-8?B?QmrDtnJuVMO2cGVs?= <bjorn.topel@intel.com>, w@1wt.eu,
        Saeed Mahameed <saeedm@mellanox.com>,
        mykyta.iziumtsev@gmail.com,
        Daniel Borkmann <borkmann@iogearbox.net>,
        Alexei Starovoitov <alexei.starovoitov@gmail.com>,
        Tariq Toukan <tariqt@mellanox.com>, brouer@redhat.com
To: Florian Westphal <fw@strlen.de>
Return-path: <netdev-owner@vger.kernel.org>
Received: from mx1.redhat.com ([209.132.183.28]:55624 "EHLO mx1.redhat.com"
        rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP
        id S1726111AbeLHLgW (ORCPT <rfc822;netdev@vger.kernel.org>);
        Sat, 8 Dec 2018 06:36:22 -0500
In-Reply-To: <20181208095758.GA32028@strlen.de>
Sender: netdev-owner@vger.kernel.org
List-ID: <netdev.vger.kernel.org>

On Sat, 8 Dec 2018 10:57:58 +0100
Florian Westphal <fw@strlen.de> wrote:

> Jesper Dangaard Brouer <brouer@redhat.com> wrote:
> > From: Ilias Apalodimas <ilias.apalodimas@linaro.org>
> > 
> > This patch is changing struct sk_buff, and is thus per-definition
> > controversial.
> > 
> > Place a new member 'mem_info' of type struct xdp_mem_info, just after
> > members (flags) head_frag and pfmemalloc, And not in between
> > headers_start/end to ensure skb_copy() and pskb_copy() work as-is.
> > Copying mem_info during skb_clone() is required.  This makes sure that
> > pages are correctly freed or recycled during the altered
> > skb_free_head() invocation.  
> 
> I read this to mean that this 'info' isn't accessed/needed until skb
> is freed.  Any reason its not added at the end?
>
> This would avoid moving other fields that are probably accessed
> more frequently during processing.

It is placed here because it is close to skb->head_frag, which is used
also used when SKB is freed.  This is the reason, both because I think
we can move the skb->head_frag bit into mem_info, and due to cacheline
accesses (e.g. skb->cloned and destructor pointer are also read, which
is on that cache-line)

The annoying part is actually that depending on the kernel config
options CONFIG_XFRM, CONFIG_NF_CONNTRACK and CONFIG_BRIDGE_NETFILTER,
whether there is a cache-line split, where mem_info gets moved into the
next cacheline.

I do like the idea of moving it to the end, iif we also move
skb->head_frag into mem_info, as skb->head (on end-cacheline) is also
accessed during free, but given we still need to read the cache-line
containing skb->{cloned,destructor}, when I don't think it will be a
win.  I think the skb->cloned value is read during lifetime of the SKB.

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer