From: Eric Dumazet <eric.dumazet@gmail.com>
To: Jonathan Lemon <jonathan.lemon@gmail.com>
Cc: netdev@vger.kernel.org, kernel-team@fb.com, axboe@kernel.dk,
Govindarajulu Varadarajan <gvaradar@cisco.com>,
Michal Kubecek <mkubecek@suse.cz>
Subject: Re: [RFC PATCH 06/21] mlx5: add header_split flag
Date: Thu, 18 Jun 2020 15:34:54 -0700 [thread overview]
Message-ID: <67362afa-45b3-d9b0-45bc-cde7eca3a550@gmail.com> (raw)
In-Reply-To: <20200618215053.qxnjegm4h5i3mvfu@bsd-mbp.dhcp.thefacebook.com>
On 6/18/20 2:50 PM, Jonathan Lemon wrote:
> On Thu, Jun 18, 2020 at 11:12:57AM -0700, Eric Dumazet wrote:
>>
>>
>> On 6/18/20 9:09 AM, Jonathan Lemon wrote:
>>> Adds a "rx_hd_split" private flag parameter to ethtool.
>>>
>>> This enables header splitting, and sets up the fragment mappings.
>>> The feature is currently only enabled for netgpu channels.
>>
>> We are using a similar idea (pseudo header split) to implement 4096+(headers) MTU at Google,
>> to enable TCP RX zerocopy on x86.
>>
>> Patch for mlx4 has not been sent upstream yet.
>>
>> For mlx4, we are using a single buffer of 128*(number_of_slots_per_RX_RING),
>> and 86 bytes for the first frag, so that the payload exactly fits a 4096 bytes page.
>>
>> (In our case, most of our data TCP packets only have 12 bytes of TCP options)
>>
>>
>> I suggest that instead of a flag, you use a tunable, that can be set by ethtool,
>> so that the exact number of bytes can be tuned, instead of hard coded in the driver.
>
> Sounds reasonable - in the long run, it would be ideal to have the
> hardware actually perform header splitting, but for now using a tunable
> fixed offset will work. In the same vein, there should be a similar
> setting for the TCP option padding on the sender side.
>
Some NIC have variable header split (Intel ixgbe I am pretty sure)
We use a mix of NIC, some with variable header splits, some with fixed pseudo header split (mlx4)
Because of this, we had to limit TCP advmss to 4108 (4096 + 12), regardless of the NIC abilities.
next prev parent reply other threads:[~2020-06-18 22:35 UTC|newest]
Thread overview: 28+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-06-18 16:09 [RFC PATCH 00/21] netgpu: networking between NIC and GPU/CPU Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 01/21] mm: add {add|release}_memory_pages Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 02/21] mm: Allow DMA mapping of pages which are not online Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 03/21] tcp: Pad TCP options out to a fixed size Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 04/21] mlx5: add definitions for header split and netgpu Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 05/21] mlx5/xsk: check that xsk does not conflict with netgpu Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 06/21] mlx5: add header_split flag Jonathan Lemon
2020-06-18 18:12 ` Eric Dumazet
2020-06-18 20:25 ` Michal Kubecek
2020-06-18 22:45 ` Eric Dumazet
2020-06-18 21:50 ` Jonathan Lemon
2020-06-18 22:34 ` Eric Dumazet [this message]
2020-06-18 22:36 ` Eric Dumazet
2020-06-18 16:09 ` [RFC PATCH 07/21] mlx5: remove the umem parameter from mlx5e_open_channel Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 08/21] misc: add shqueue.h for prototyping Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 09/21] include: add definitions for netgpu Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 10/21] mlx5: add netgpu queue functions Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 11/21] skbuff: add a zc_netgpu bitflag Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 12/21] mlx5: hook up the netgpu channel functions Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 13/21] netdevice: add SETUP_NETGPU to the netdev_bpf structure Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 14/21] kernel: export free_uid Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 15/21] netgpu: add network/gpu dma module Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 16/21] lib: have __zerocopy_sg_from_iter get netgpu pages for a sk Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 17/21] net/core: add the SO_REGISTER_DMA socket option Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 18/21] tcp: add MSG_NETDMA flag for sendmsg() Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 19/21] core: add page recycling logic for netgpu pages Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 20/21] core/skbuff: use skb_zdata for testing whether skb is zerocopy Jonathan Lemon
2020-06-18 16:09 ` [RFC PATCH 21/21] mlx5: add XDP_SETUP_NETGPU hook Jonathan Lemon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=67362afa-45b3-d9b0-45bc-cde7eca3a550@gmail.com \
--to=eric.dumazet@gmail.com \
--cc=axboe@kernel.dk \
--cc=gvaradar@cisco.com \
--cc=jonathan.lemon@gmail.com \
--cc=kernel-team@fb.com \
--cc=mkubecek@suse.cz \
--cc=netdev@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).