RE: [PATCH V1 net-next 0/5] Add devlink support to ena

["Arinzon, David" <darinzon@amazon.com>]

> -----Original Message-----
> From: Jakub Kicinski <kuba@kernel.org>
> Sent: Wednesday, January 11, 2023 9:01 PM
> To: Arinzon, David <darinzon@amazon.com>
> Cc: David Miller <davem@davemloft.net>; netdev@vger.kernel.org;
> Machulsky, Zorik <zorik@amazon.com>; Matushevsky, Alexander
> <matua@amazon.com>; Bshara, Saeed <saeedb@amazon.com>; Bshara,
> Nafea <nafea@amazon.com>; Saidi, Ali <alisaidi@amazon.com>;
> Kiyanovski, Arthur <akiyano@amazon.com>; Dagan, Noam
> <ndagan@amazon.com>; Agroskin, Shay <shayagr@amazon.com>; Itzko,
> Shahar <itzko@amazon.com>; Abboud, Osama <osamaabb@amazon.com>
> Subject: RE: [EXTERNAL][PATCH V1 net-next 0/5] Add devlink support to
> ena
> 
> CAUTION: This email originated from outside of the organization. Do not
> click links or open attachments unless you can confirm the sender and
> know the content is safe.
> 
> 
> 
> On Wed, 11 Jan 2023 08:58:46 +0000 Arinzon, David wrote:
> > > I read it again - and I still don't know what you're doing.
> > > I sounds like inline header length configuration yet you also use
> > > LLQ all over the place. And LLQ for ENA is documented as basically
> tx_push:
> > >
> > >   - **Low Latency Queue (LLQ) mode or "push-mode":**
> > >
> > > Please explain this in a way which assumes zero Amazon-specific
> > > knowledge :(
> >
> > Low Latency Queues (LLQ) is a mode of operation where the packet
> > headers (up to a defined length) are being written directly to the device
> memory.
> > Therefore, you are right, the description is similar to tx_push.
> > However, This is not a configurable option while
> > ETHTOOL_A_RINGS_TX_PUSH configures whether to work in a mode or
> not.
> > If I'm understanding the intent behind ETHTOOL_A_RINGS_TX_PUSH
> and the
> > implementation in the driver that introduced the feature, it refers to
> > a push of the packet and not just the headers, which is not what the
> > ena driver does.
> >
> > In this patchset, we allow the configuration of an extended size of
> > the Low Latency Queue, meaning, allow enabled another, larger,
> > pre-defined size to be used as a max size of the packet header to be
> > pushed directly to device memory. It is not configurable in value,
> > therefore, it was defined as large LLQ.
> >
> > I hope this provides more clarification, if not, I'll be happy to elaborate
> further.
> 
> Thanks, the large missing piece in my understanding is still what the user
> visible impact of this change is. Without increasing the LLQ entry size, a
> user who sends packet with long headers will:
>  a) see higher latency thru the NIC, but everything else is the same
>  b) see higher latency and lower overall throughput in terms of PPS
>  c) will have limited access to offloads, because the device requires
>     full access to headers via LLQ for some offloads
> 
> which one of the three is the closest?

You're right, I went through the documentation again, and I see that the implications
of a case where packet headers are longer than the LLQ entry size are not mentioned
properly. We'll rework it to explain the motivation to turn on this mode in relevant use cases.
If the packet network headers are not within the size of the LLQ entry, then the packet will
be dropped. So I'll say that c) describes the impact the best given that certain types of
traffic will not succeed or have disruptions due to dropped TX packets.