Re: [PATCH V1 net-next 0/5] Add devlink support to ena

[Jakub Kicinski <kuba@kernel.org>]

On Tue, 10 Jan 2023 20:11:23 +0000 Arinzon, David wrote:
> > On Sun, 8 Jan 2023 10:35:28 +0000 David Arinzon wrote:  
> > > This patchset adds devlink support to the ena driver.  
> > 
> > Wrong place, please take a look at
> > 
> >         struct kernel_ethtool_ringparam::tx_push
> > 
> > and ETHTOOL_A_RINGS_TX_PUSH. I think you just want to configure the
> > max size of the TX push, right?
> 
> We're not configuring the max size of the TX push, but effectively the
> maximal packet header size to be pushed to the device.
> This is noted in the documentation on patch 5/5 in this patchset.
> AFAIK, there's no relevant ethtool parameter for this configuration.

Perhaps I should have complained about the low quality of that
documentation to make it clear that I have in fact read it :/

I read it again - and I still don't know what you're doing.
I sounds like inline header length configuration yet you also use LLQ
all over the place. And LLQ for ENA is documented as basically tx_push:

  - **Low Latency Queue (LLQ) mode or "push-mode":**

Please explain this in a way which assumes zero Amazon-specific
knowledge :(

> > The reload is also an overkill, reload should re-register all driver objects
> > but the devlink instance, IIRC. You're not even unregistering the netdev.
> > You should handle this change the same way you handle any ring size
> > changes.
> 
> The LLQ configuration is different from other configurations set via ethtool
> (like queue size and number of queues). LLQ requires re-negotiation
> with the device and requires a reset, which is not performed in the ethtool
> configurations case.

What do you mean when you say that reset is not required in the ethool
configuration case?

AFAIK ethtool config should not (although sadly very often it does)
cause any loss of unrelated configuration. But you can certainly reset
HW blocks or reneg features with FW or whatever else...

> It may be possible to unregister/register the netdev,
> but it is unnecessary in this case, as most of the changes are reflected in the
> interface and structures between the driver and the device.
> 
> > For future reference - if you ever _actually_ need devlink please use the
> > devl_* APIs and take the instance locks explicitly. There has not been a
> > single devlink reload implementation which would get locking right using
> > the devlink_* APIs 😔️  
> 
> This operation can happen in parallel to a reset of the device from a different
> context which is unrelated to devlink. Our intention is to avoid such cases,
> therefore, holding the devlink lock using devl_lock APIs will not be sufficient.
> The driver holds the RTNL_LOCK in key places, either explicitly or implicitly,
> as in ethtool configuration changes for example.

Yeah, which is why you should not be using devlink for this.