Possibility of me mainlining Tehuti Networks 10GbE driver

Possibility of me mainlining Tehuti Networks 10GbE driver

[Nicholas Johnson]

Hi all,

To start off, if I am emailing the wrong people, please blame the output 
of: "scripts/get_maintainer.pl drivers/net/ethernet/tehuti/" and let me 
know who I should be contacting. Should I add in 
"linux-kernel@vger.kernel.org"?

I just discovered that the Tehuti 10GbE networking drivers (required for 
things such as some AKiTiO Thunderbolt to 10GbE adapters) are not in 
mainline. I am interested in mainlining it, but need to know how much 
work it would take and if it will force me to be the maintainer for all 
eternity.

The driver, in tn40xx-0.3.6.15-c.tar appears to be available here:
Link: https://www.akitio.com/faq/341-thunder3-10gbe-adapter-can-i-use-this-network-adapter-on-linux
Also here:
Link: https://github.com/acooks/tn40xx-driver

I see some immediate style problems and indentation issues. I can fix 
these.

The current driver only works with Linux v4.19, I believe. There are a 
small handful of compile errors with v5.4. I can probably fix these.

However, could somebody please comment on any technical issues that you 
can see here? How much work do you think I would have to do to mainline 
this? Would I have to buy such a device for testing? Would I have to buy 
*all* of the supported devices for testing? Or can other people do that 
for me?

I am not keen on having to buy anything without mainline support - it is 
an instant disqualification of a hardware vendor. It results in a 
terrible user experience for experienced people (might not be able to 
use latest kernel which is needed for supporting other things) and is 
debilitating for people new to Linux who do not how to use the terminal, 
possibly enough so that they will go back to Windows.

Andy, what is your relationship to Tehuti Networks? Would you be happy 
to maintain this if I mainlined it? It says you are maintainer of 
drivers/net/ethernet/tehuti/ directory. I will not do this if I am 
expected to maintain it - in no small part because I do not know a lot 
about it. I will only be modifying what is currently available to make 
it acceptable for mainline, if possible.

Also, license issues - does GPLv2 permit mainlining to happen? I believe 
the Tehuti driver is available under GPLv2 (correct me if I am wrong).

Would I need to send patches for this, or for something this size, is it 
better to send a pull request? If I am going to do patches, I will need 
to make a gmail account or something, as Outlook messes with the 
encoding of the things which I send.

Thanks for any comments on this.

Kind regards,
Nicholas Johnson

Re: Possibility of me mainlining Tehuti Networks 10GbE driver

[Florian Fainelli]

Hi,

On 11/7/2019 6:24 PM, Nicholas Johnson wrote:
> Hi all,
> 
> To start off, if I am emailing the wrong people, please blame the output 
> of: "scripts/get_maintainer.pl drivers/net/ethernet/tehuti/" and let me 
> know who I should be contacting. Should I add in 
> "linux-kernel@vger.kernel.org"?
> 
> I just discovered that the Tehuti 10GbE networking drivers (required for 
> things such as some AKiTiO Thunderbolt to 10GbE adapters) are not in 
> mainline. I am interested in mainlining it, but need to know how much 
> work it would take and if it will force me to be the maintainer for all 
> eternity.
> 
> The driver, in tn40xx-0.3.6.15-c.tar appears to be available here:
> Link: https://www.akitio.com/faq/341-thunder3-10gbe-adapter-can-i-use-this-network-adapter-on-linux
> Also here:
> Link: https://github.com/acooks/tn40xx-driver
> 
> I see some immediate style problems and indentation issues. I can fix 
> these.
> 
> The current driver only works with Linux v4.19, I believe. There are a 
> small handful of compile errors with v5.4. I can probably fix these.
> 
> However, could somebody please comment on any technical issues that you 
> can see here? How much work do you think I would have to do to mainline 
> this? Would I have to buy such a device for testing? Would I have to buy 
> *all* of the supported devices for testing? Or can other people do that 
> for me?

This is based on roughly 5 minutes of browsing source files, but what I
see, which is typical from out of tree vendor drivers is a complete lack
of use of existing kernel APIs beyond registering a net_device which you
would have to use to seek upstream inclusion, that includes for the most
part:

- make use of the PHYLINK subsystem for supporting 10GBaseT and SFP
modules instead of doing your own, there might be existing PHY drivers
that you can use for the Aquantia and Marvell parts, see
drivers/net/phy/ to check whether the PHY models are indeed supported

- implement a proper mii_bus interface for "talking" to the PHYs,
implement a proper gpio_chip instance to register with Linux as a GPIO
controller, such that then you can use i2c-gpio to become an i2c bus
master driver, and then talk to the SFPs properly

- lots and lots of stylistic issues that must be fixed

- getting rid of private driver ioctl implementation

There are certainly many more details once we start digging of course.

> 
> I am not keen on having to buy anything without mainline support - it is 
> an instant disqualification of a hardware vendor. It results in a 
> terrible user experience for experienced people (might not be able to 
> use latest kernel which is needed for supporting other things) and is 
> debilitating for people new to Linux who do not how to use the terminal, 
> possibly enough so that they will go back to Windows.

Seems like a reasonable position to me, the grey area is when there is a
Linux driver, but its quality is not making it upstream available, then
you find yourself emailing netdev about that very situation :)

> 
> Andy, what is your relationship to Tehuti Networks? Would you be happy 
> to maintain this if I mainlined it? It says you are maintainer of 
> drivers/net/ethernet/tehuti/ directory. I will not do this if I am 
> expected to maintain it - in no small part because I do not know a lot 
> about it. I will only be modifying what is currently available to make 
> it acceptable for mainline, if possible.

Given how the driver is broken up, you can do a couple of strategies:

- try to submit it all as-is (almost) under drivers/staging/ where it
may get contributions from people to clean it up to the kernel coding
style, using coccinelle semantic patch and pretty much anything that can
be done by inspecting code visually while not really testing it. This
might make the driver stay in staging for a long time, but if there are
in-kernel API changes, they will be done and so it will continue to
build and maybe even work, for any version of Linux in which it got
included and onward. The problem with that approach is that it will
likely stay in limbo unless a dedicated set of people start working
towards moving it out of staging.

- rewrite it in smaller parts and submit it in small chunks, with basic
functionality one step at a time, e.g.: driver skeleton/entry point as a
pci_device/driver, then net_device registration without anything, then
RX path, then TX path, then control path, then ethtool interface, etc.
etc. Given the shape of the driver, but not knowing how familiar you are
with the driver or the kernel, a 3 man/month work for someone motivated
is probably an optimistic estimate of the work you have ahead of you,
6m/m sounds more realistic. There is also an expectation that you will
be maintaining this driver for a few months (maybe years) to come, and
network drivers tend to always have something that needs to be fixed, so
it is a nice side gig, but it could be time consuming.

> 
> Also, license issues - does GPLv2 permit mainlining to happen? I believe 
> the Tehuti driver is available under GPLv2 (correct me if I am wrong).

The source code on the github tree suggests this is the case, therefore
it is entirely appropriate to seek upstream inclusion given the license
allows it.

What needs to be figured out is the PHY firmware situation which appears
to be completed punted onto the user to figure out which files (and
where to download), how to extract the relevant firmware blobs (there
are scripts, okay). If you have a contact with one of the vendors
supported by the driver, or better yet, with Tehuti, that may be
something they could help with. A mainline driver with proprietary
firmware blobs is not uncommon, but having to get the blobs outside of
linux-firmware is a real pain for distributions and some might even
refuse to build your driver because of that.

> 
> Would I need to send patches for this, or for something this size, is it 
> better to send a pull request? If I am going to do patches, I will need 
> to make a gmail account or something, as Outlook messes with the 
> encoding of the things which I send.

For sending patches, you would want to use git send-email to make sure
you avoid MUA issues.

> 
> Thanks for any comments on this.

Hope all of this helps. Cheers
-- 
Florian

Re: Possibility of me mainlining Tehuti Networks 10GbE driver

[Nicholas Johnson]

On Thu, Nov 07, 2019 at 08:06:30PM -0800, Florian Fainelli wrote:
> Hi,
> 
> On 11/7/2019 6:24 PM, Nicholas Johnson wrote:
> > Hi all,
> > 
> > To start off, if I am emailing the wrong people, please blame the output 
> > of: "scripts/get_maintainer.pl drivers/net/ethernet/tehuti/" and let me 
> > know who I should be contacting. Should I add in 
> > "linux-kernel@vger.kernel.org"?
> > 
> > I just discovered that the Tehuti 10GbE networking drivers (required for 
> > things such as some AKiTiO Thunderbolt to 10GbE adapters) are not in 
> > mainline. I am interested in mainlining it, but need to know how much 
> > work it would take and if it will force me to be the maintainer for all 
> > eternity.
> > 
> > The driver, in tn40xx-0.3.6.15-c.tar appears to be available here:
> > Link: https://www.akitio.com/faq/341-thunder3-10gbe-adapter-can-i-use-this-network-adapter-on-linux
> > Also here:
> > Link: https://github.com/acooks/tn40xx-driver
> > 
> > I see some immediate style problems and indentation issues. I can fix 
> > these.
> > 
> > The current driver only works with Linux v4.19, I believe. There are a 
> > small handful of compile errors with v5.4. I can probably fix these.
> > 
> > However, could somebody please comment on any technical issues that you 
> > can see here? How much work do you think I would have to do to mainline 
> > this? Would I have to buy such a device for testing? Would I have to buy 
> > *all* of the supported devices for testing? Or can other people do that 
> > for me?
> 
> This is based on roughly 5 minutes of browsing source files, but what I
> see, which is typical from out of tree vendor drivers is a complete lack
> of use of existing kernel APIs beyond registering a net_device which you
> would have to use to seek upstream inclusion, that includes for the most
> part:
> 
> - make use of the PHYLINK subsystem for supporting 10GBaseT and SFP
> modules instead of doing your own, there might be existing PHY drivers
> that you can use for the Aquantia and Marvell parts, see
> drivers/net/phy/ to check whether the PHY models are indeed supported
> 
> - implement a proper mii_bus interface for "talking" to the PHYs,
> implement a proper gpio_chip instance to register with Linux as a GPIO
> controller, such that then you can use i2c-gpio to become an i2c bus
> master driver, and then talk to the SFPs properly
> 
> - lots and lots of stylistic issues that must be fixed
> 
> - getting rid of private driver ioctl implementation
> 
> There are certainly many more details once we start digging of course.
> 
> > 
> > I am not keen on having to buy anything without mainline support - it is 
> > an instant disqualification of a hardware vendor. It results in a 
> > terrible user experience for experienced people (might not be able to 
> > use latest kernel which is needed for supporting other things) and is 
> > debilitating for people new to Linux who do not how to use the terminal, 
> > possibly enough so that they will go back to Windows.
> 
> Seems like a reasonable position to me, the grey area is when there is a
> Linux driver, but its quality is not making it upstream available, then
> you find yourself emailing netdev about that very situation :)
To clarify, I do not own the device, but I realised that the driver 
needs mainlining and this could be good experience.

> 
> > 
> > Andy, what is your relationship to Tehuti Networks? Would you be happy 
> > to maintain this if I mainlined it? It says you are maintainer of 
> > drivers/net/ethernet/tehuti/ directory. I will not do this if I am 
> > expected to maintain it - in no small part because I do not know a lot 
> > about it. I will only be modifying what is currently available to make 
> > it acceptable for mainline, if possible.
> 
> Given how the driver is broken up, you can do a couple of strategies:
> 
> - try to submit it all as-is (almost) under drivers/staging/ where it
> may get contributions from people to clean it up to the kernel coding
> style, using coccinelle semantic patch and pretty much anything that can
> be done by inspecting code visually while not really testing it. This
> might make the driver stay in staging for a long time, but if there are
> in-kernel API changes, they will be done and so it will continue to
> build and maybe even work, for any version of Linux in which it got
> included and onward. The problem with that approach is that it will
> likely stay in limbo unless a dedicated set of people start working
> towards moving it out of staging.
> 
> - rewrite it in smaller parts and submit it in small chunks, with basic
> functionality one step at a time, e.g.: driver skeleton/entry point as a
> pci_device/driver, then net_device registration without anything, then
> RX path, then TX path, then control path, then ethtool interface, etc.
> etc. Given the shape of the driver, but not knowing how familiar you are
> with the driver or the kernel, a 3 man/month work for someone motivated
> is probably an optimistic estimate of the work you have ahead of you,
> 6m/m sounds more realistic. There is also an expectation that you will
> be maintaining this driver for a few months (maybe years) to come, and
> network drivers tend to always have something that needs to be fixed, so
> it is a nice side gig, but it could be time consuming.
Okay, that is a lot. Perhaps this is unwise.

One strategy could be to hollow out a mainline driver and use its 
structure?

> 
> > 
> > Also, license issues - does GPLv2 permit mainlining to happen? I believe 
> > the Tehuti driver is available under GPLv2 (correct me if I am wrong).
> 
> The source code on the github tree suggests this is the case, therefore
> it is entirely appropriate to seek upstream inclusion given the license
> allows it.
> 
> What needs to be figured out is the PHY firmware situation which appears
> to be completed punted onto the user to figure out which files (and
> where to download), how to extract the relevant firmware blobs (there
> are scripts, okay). If you have a contact with one of the vendors
> supported by the driver, or better yet, with Tehuti, that may be
> something they could help with. A mainline driver with proprietary
> firmware blobs is not uncommon, but having to get the blobs outside of
> linux-firmware is a real pain for distributions and some might even
> refuse to build your driver because of that.
Could I please have more information / reading resources on the PHY 
business? My understanding is that the NIC firmware would be dealing 
with the PHY (this is the MII, right?) and the OS would have nothing to 
do with it (it just sees a NIC which processes packets). But it seems 
like this is not the case. Can you please clarify this? One guess is 
that the NIC firmware handles most of it, but the OS is able to detect 
and display info on the PHY to userspace. Incorrect?

Above you mentioned mii_bus for PHYs, but a driver like Aquantia 
Atlantic does not have references to PHYs or MII. Why do some not need 
the feature when others do?

> 
> > 
> > Would I need to send patches for this, or for something this size, is it 
> > better to send a pull request? If I am going to do patches, I will need 
> > to make a gmail account or something, as Outlook messes with the 
> > encoding of the things which I send.
> 
> For sending patches, you would want to use git send-email to make sure
> you avoid MUA issues.
> 
> > 
> > Thanks for any comments on this.
> 
> Hope all of this helps. Cheers
Thanks, it does answer a lot of questions.

> -- 
> Florian

Would it make it easier to buy a single Tehuti NIC and try to make a 
driver just for that model? Or does each additional model generally not 
add much more work?

I would dive into this and see how far I can get, but I have to make an 
investment in an expensive piece of hardware I do not particularly want.

Given my inexperience, it is unlikely to succeed, and then I have a 
device laying around without mainline support, making it difficult to 
use.

Perhaps a better tree to be barking up is "what do I need to do to get 
to the point where I could do something like this?" Can you suggest 
smaller pieces of work which are not trivial / harder than code cleanup 
to start building experience?

Thanks!

Regards,
Nicholas.

Re: Possibility of me mainlining Tehuti Networks 10GbE driver

[Andrew Lunn]

> Could I please have more information / reading resources on the PHY 
> business? My understanding is that the NIC firmware would be dealing 
> with the PHY (this is the MII, right?) and the OS would have nothing to 
> do with it (it just sees a NIC which processes packets).

There are two different strategies boards can use:

1) The MAC firmware handles the PHY, and the firmware has all the PHY
   drivers needed.

2) Linux handles the PHY, and uses drivers from drivers/net/phy. The
   MAC driver then needs to export an MDIO bus, and use either phylink
   or phylib.

> Above you mentioned mii_bus for PHYs, but a driver like Aquantia 
> Atlantic does not have references to PHYs or MII. Why do some not need 
> the feature when others do?

The Aquantia MAC driver uses the firmware strategy. The PHY is
completely hidden from Linux with the firmware running on the MAC
handling the PHY.

But say look at the Marvell MAC drivers. They expose an MDIO bus, and
linux then drives the PHY. The MAC driver then uses the phylink_ API
to interact with the PHY via the phylink core code.

I also had a quick look at the code. It has the basic code you need to
export an MDIO bus, and let Linux control the PHYs.  bdx_mdio_read()
and bdx_mdio_write() would become the two functions you need for
struct mii_bus, and then pass it to mdiobus_register(). You would then
need to add phylink calls in the write place in the MAC driver.

If you think writing a new driver is too much work for you, then going
via staging is probably the better idea. You can slowly working on
improving the driver. One of the big chunks of work is going to be
swapping to the kernel PHY drivers, using PHYLINK, etc. And to do
that, you really do need to have hardware. You don't need all the
different variants the driver supports, but you should at least have
one. And i would suggest you get hardware which uses either the
Marvell or Aquantia PHY.

     Andrew

Re: Possibility of me mainlining Tehuti Networks 10GbE driver

[Andy Gospodarek]

On Fri, Nov 08, 2019 at 02:24:44AM +0000, Nicholas Johnson wrote:
> Hi all,
> 
> To start off, if I am emailing the wrong people, please blame the output 
> of: "scripts/get_maintainer.pl drivers/net/ethernet/tehuti/" and let me 
> know who I should be contacting. Should I add in 
> "linux-kernel@vger.kernel.org"?
> 
> I just discovered that the Tehuti 10GbE networking drivers (required for 
> things such as some AKiTiO Thunderbolt to 10GbE adapters) are not in 
> mainline. I am interested in mainlining it, but need to know how much 
> work it would take and if it will force me to be the maintainer for all 
> eternity.
> 
> The driver, in tn40xx-0.3.6.15-c.tar appears to be available here:
> Link: https://www.akitio.com/faq/341-thunder3-10gbe-adapter-can-i-use-this-network-adapter-on-linux
> Also here:
> Link: https://github.com/acooks/tn40xx-driver
> 
> I see some immediate style problems and indentation issues. I can fix 
> these.
> 
> The current driver only works with Linux v4.19, I believe. There are a 
> small handful of compile errors with v5.4. I can probably fix these.
> 
> However, could somebody please comment on any technical issues that you 
> can see here? How much work do you think I would have to do to mainline 
> this? Would I have to buy such a device for testing? Would I have to buy 
> *all* of the supported devices for testing? Or can other people do that 
> for me?
> 
> I am not keen on having to buy anything without mainline support - it is 
> an instant disqualification of a hardware vendor. It results in a 
> terrible user experience for experienced people (might not be able to 
> use latest kernel which is needed for supporting other things) and is 
> debilitating for people new to Linux who do not how to use the terminal, 
> possibly enough so that they will go back to Windows.
> 
> Andy, what is your relationship to Tehuti Networks? Would you be happy 
> to maintain this if I mainlined it? It says you are maintainer of 
> drivers/net/ethernet/tehuti/ directory. I will not do this if I am 
> expected to maintain it - in no small part because I do not know a lot 
> about it. I will only be modifying what is currently available to make 
> it acceptable for mainline, if possible.

[Nicolas, sorry for the slow response -- I've been AFK for a bit.]

A long time ago, in a galaxy far, far away Tehuti sent me one of their
early 10GbE adapters and asked if I would help them take their driver
upstream.  They provided an out of tree driver as a basis and after a
few revisions David Miller agreed to take this into net-next.  The
driver as it exists today could use lots of work.  There were many items
on my TODO list for it, but I never made the time to clean it up
properly so it could still use some care and feeding.  I just checked my
cache of cards and unfortunately it looks like I do not have any of
these adapters at home any longer.  I may need to check the office to
see if I have one there, but I think chances are slim.

I'd feel better about helping to maintain the driver if there was
hardware available for whoever was doing the work.  It looks like there
are some pretty cheap (sub-200USD) cards available online that use that
chipset.  Frankly, I'd probably also feel better about maintaining it
and updating to all the coolest new features if I didn't currently work
at another hardware vendor, so I need to consider that.

I haven't pulled down their latest driver from github, but I'd be
curious to see how close the hardware drivers appear to be between the
40xx chipset and the original TOE SmartNIC[sic] that is supported
upstream today.  Did you by any chance compare the two?

> Also, license issues - does GPLv2 permit mainlining to happen? I believe 
> the Tehuti driver is available under GPLv2 (correct me if I am wrong).
> 
> Would I need to send patches for this, or for something this size, is it 
> better to send a pull request? If I am going to do patches, I will need 
> to make a gmail account or something, as Outlook messes with the 
> encoding of the things which I send.
> 
> Thanks for any comments on this.
> 
> Kind regards,
> Nicholas Johnson