Re: [net-next PATCH 00/15] eth: fbnic: Add network driver for Meta Platforms Host Network Interface

[Alexander Duyck <alexander.duyck@gmail.com>]

On Thu, Apr 11, 2024 at 10:32 AM Andrew Lunn <andrew@lunn.ch> wrote:
>
> On Thu, Apr 11, 2024 at 09:00:17AM -0700, Alexander Duyck wrote:
> > On Wed, Apr 10, 2024 at 3:37 PM Andrew Lunn <andrew@lunn.ch> wrote:
> > >
> > > > Well I was referring more to the data path level more than the phy
> > > > configuration. I suspect different people have different levels of
> > > > expectations on what minimal firmware is. With this hardware we at
> > > > least don't need to use firmware commands to enable or disable queues,
> > > > get the device stats, or update a MAC address.
> > > >
> > > > When it comes to multi-host NICs I am not sure there are going to be
> > > > any solutions that don't have some level of firmware due to the fact
> > > > that the cable is physically shared with multiple slots.
> > >
> > > This is something Russell King at least considered. I don't really
> > > know enough to know why its impossible for Linux to deal with multiple
> > > slots.
> >
> > It mostly has to do with the arbitration between them. It is a matter
> > of having to pass a TON of info to the individual slice and then the
> > problem is it would have to do things correctly and not manage to take
> > out it's neighbor or the BMC.
>
> How much is specific to your device? How much is just following 802.3
> and the CMIS standards? I assume anything which is just following
> 802.3 and CMIS could actually be re-used? And you have some glue to
> combine them in a way that is specific to your device?
>
> > > > I am assuming we still want to do the PCS driver. So I will still see
> > > > what I can do to get that setup.
> > >
> > > You should look at the API offered by drivers in drivers/net/pcs. It
> > > is designed to be used with drivers which actually drive the hardware,
> > > and use phylink. Who is responsible for configuring and looking at the
> > > results of auto negotiation? Who is responsible for putting the PCS
> > > into the correct mode depending on the SFP modules capabilities?
> > > Because you seemed to of split the PCS into two, and hidden some of it
> > > away, i don't know if it makes sense to try to shoehorn what is left
> > > into a Linux driver.
> >
> > We have control of the auto negotiation as that is north of the PMA
> > and is configured per host. We should support clause 73 autoneg.
> > Although we haven't done much with it as most of our use cases are
> > just fixed speed setups to the switch over either 25G-CR1, 50G-CR2,
> > 50G-CR1, or 100G-CR2. So odds are we aren't going to be doing anything
> > too terribly exciting.
>
> Maybe not, but you might of gained from the community here, if others
> could of adopted this code for their devices. You might not need
> clause 73, but phylink provides helpers to implement it, so it is
> pretty easy to add. Maybe your initial PCS driver does not support it,
> but later adopters who also licence this PCS might add it, and you get
> the feature for free. The corrected/uncorrected counters i asked
> about, are something you might not export in your current code via
> ethtool. But again, this is something which somebody else could add a
> helper for, and you would get it nearly for free.

You don't have to sell me on the reuse advantages of open source. I
will probably look at adding autoneg at some point in the future, but
for our main use case it wasn't needed. If nothing else I will
probably hand it off to one of the new hires on the team when I get
some time.

The counters are exported. Just haven't gotten far enough to show the
ethtool patches yet.. :-)

> > As far as the QSFP setup the FW is responsible for any communication
> > with it. I suspect that the expectation is that we aren't going to
> > need much in the way of config since we are just using direct attach
> > cables.
>
> Another place you might of got features for free. The Linux SFP driver
> exports HWMON values for temperature, power, received power, etc, but
> for 1G. The QSFP+ standard Versatile Diagnostics Monitoring is
> different, but i could see somebody adding a generic implementation in
> the Linux SFP driver, so that the HWMON support is just free. Same
> goes for the error performance statics. Parts of power management
> could easily be generic. It might be possible to use Linux regulators
> to describe what your board is capable if, and the SFP core could then
> implement the ethtool ops, checking with the regulator to see if the
> power is actually available, and then talking to the SFP to tell it to
> change its power class?

Again, for us it ends up not having much value adding additional QSFP
logic because we aren't using anything fancy. It is all just direct
attach cables.

> Florian posted some interesting statistics, that vendors tend to
> maintain their own drivers, and don't get much support from the
> community. However I suspect it is a different story for shared
> infrastructure like PCS drivers, PHY drivers, SFP drivers. That is
> where you get the most community support and the most stuff for free.
> But you actually have to use it to benefit from it.

I'll probably get started on the PCS drivers for this next week. I
will follow up with questions if I run into any issues.

Thanks,

- Alex