DSA driver - how to glue to a PCI based NIC's mdio?

DSA driver - how to glue to a PCI based NIC's mdio?

[Tim Harvey]

Greetings,

I'm working on adding DSA support for a PCIe expansion card (designed
by us) that has common PCIe NIC connected via its mii-bus to a Marvell
MV88E6171. Because the NIC is a PCIe device, it has no device-tree
representation of its NIC or its mdio bus, but does register its mdio
bus with Linux.

Does anyone know how I would be able to specify the Ethernet device
and MDIO bus for this add-in card if it has no device-tree handle?

I can write a phy driver that gets probed by matching the MV88E6171
PHY ID when the NIC's mdio bus get's registered yet even then I'm not
clear how to get hold of the netdev pointer given the mdio bus so as
to build a dsa_chip_data struct to register as a platform device. I'm
also not clear in this case how to verify in this case if the
MV88E6171 is from 'my' add-in card vs another card.

Perhaps the right approach is to program the NIC's EEPROM on our board
with a PCI_ID/DEVICE_ID of ours, add support for those ID's to the
NIC's driver, and within the NIC's driver create and register dsa
platform device when our ID is encountered?

Thanks for any advise,

Tim

Tim Harvey - Principal Software Engineer
Gateworks Corporation - http://www.gateworks.com/
3026 S. Higuera St. San Luis Obispo CA 93401
805-781-2000

Re: DSA driver - how to glue to a PCI based NIC's mdio?

[Andrew Lunn]

On Wed, Sep 30, 2015 at 01:44:52PM -0700, Tim Harvey wrote:
> Greetings,
> 
> I'm working on adding DSA support for a PCIe expansion card (designed
> by us) that has common PCIe NIC connected via its mii-bus to a Marvell
> MV88E6171. Because the NIC is a PCIe device, it has no device-tree
> representation of its NIC or its mdio bus, but does register its mdio
> bus with Linux.

It is possible to represent PCIe devices in device tree. Take a look
at ePAPR. Is the PCIe host in DT?

> Perhaps the right approach is to program the NIC's EEPROM on our board
> with a PCI_ID/DEVICE_ID of ours, add support for those ID's to the
> NIC's driver, and within the NIC's driver create and register dsa
> platform device when our ID is encountered?

This sounds sensible. But i doubt you can add your DSA platform
information to the NIC's device driver. Better would be to have a
small shim driver which is loaded on your PCI_ID/DEVICE_ID. That would
instantiate the NIC driver, and insert a DSA platform device.

    Andrew

Re: DSA driver - how to glue to a PCI based NIC's mdio?

[Tim Harvey]

On Wed, Sep 30, 2015 at 2:12 PM, Andrew Lunn <andrew@lunn.ch> wrote:
> On Wed, Sep 30, 2015 at 01:44:52PM -0700, Tim Harvey wrote:
>> Greetings,
>>
>> I'm working on adding DSA support for a PCIe expansion card (designed
>> by us) that has common PCIe NIC connected via its mii-bus to a Marvell
>> MV88E6171. Because the NIC is a PCIe device, it has no device-tree
>> representation of its NIC or its mdio bus, but does register its mdio
>> bus with Linux.
>
> It is possible to represent PCIe devices in device tree. Take a look
> at ePAPR. Is the PCIe host in DT?

It is possible to represent PCI devices in device-tree however not in
a dynamic or plug-able fashion - they have to be nested per bus/slot
which defeats the purpose of dynamic enumeration.

>
>> Perhaps the right approach is to program the NIC's EEPROM on our board
>> with a PCI_ID/DEVICE_ID of ours, add support for those ID's to the
>> NIC's driver, and within the NIC's driver create and register dsa
>> platform device when our ID is encountered?
>
> This sounds sensible. But i doubt you can add your DSA platform
> information to the NIC's device driver. Better would be to have a
> small shim driver which is loaded on your PCI_ID/DEVICE_ID. That would
> instantiate the NIC driver, and insert a DSA platform device.

I was thinking of this as well, but then I would still need that shim
to know the netdevice that the driver I'm shimming creates so I can't
figure a way to do it without touching the PCI driver.

Tim

Re: DSA driver - how to glue to a PCI based NIC's mdio?

[Florian Fainelli]

On 30/09/15 14:27, Tim Harvey wrote:
> On Wed, Sep 30, 2015 at 2:12 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> On Wed, Sep 30, 2015 at 01:44:52PM -0700, Tim Harvey wrote:
>>> Greetings,
>>>
>>> I'm working on adding DSA support for a PCIe expansion card (designed
>>> by us) that has common PCIe NIC connected via its mii-bus to a Marvell
>>> MV88E6171. Because the NIC is a PCIe device, it has no device-tree
>>> representation of its NIC or its mdio bus, but does register its mdio
>>> bus with Linux.
>>
>> It is possible to represent PCIe devices in device tree. Take a look
>> at ePAPR. Is the PCIe host in DT?
> 
> It is possible to represent PCI devices in device-tree however not in
> a dynamic or plug-able fashion - they have to be nested per bus/slot
> which defeats the purpose of dynamic enumeration.

Even though a bus is completely auto-discoverable, if there is
additional information needed to supplement that topology, having things
be represented in Device Tree is typically accepted.

> 
>>
>>> Perhaps the right approach is to program the NIC's EEPROM on our board
>>> with a PCI_ID/DEVICE_ID of ours, add support for those ID's to the
>>> NIC's driver, and within the NIC's driver create and register dsa
>>> platform device when our ID is encountered?
>>
>> This sounds sensible. But i doubt you can add your DSA platform
>> information to the NIC's device driver. Better would be to have a
>> small shim driver which is loaded on your PCI_ID/DEVICE_ID. That would
>> instantiate the NIC driver, and insert a DSA platform device.
> 
> I was thinking of this as well, but then I would still need that shim
> to know the netdevice that the driver I'm shimming creates so I can't
> figure a way to do it without touching the PCI driver.

You can register a network device notifier, and try to extract that
information about this network device you need once you see that device
being registered. As an example, there is a loopback/fake DSA switch
driver here which uses the loopback interface as a parent network device
(NB: this is using the network device name, which is pretty lame, but
that does the job):

https://github.com/ffainelli/linux/commit/67d1db45d17f8cc3b32d7a46c49d5df736cee56c
-- 
Florian

Re: DSA driver - how to glue to a PCI based NIC's mdio?

[Andrew Lunn]

> > information to the NIC's device driver. Better would be to have a
> > small shim driver which is loaded on your PCI_ID/DEVICE_ID. That would
> > instantiate the NIC driver, and insert a DSA platform device.
> 
> I was thinking of this as well, but then I would still need that shim
> to know the netdevice that the driver I'm shimming creates so I can't
> figure a way to do it without touching the PCI driver.

You can probably get that from the PCI information you have:

# /sys/devices/pci0000:00/0000:00:1c.2/0000:06:00.0 is the PCIe device. And it has

# /sys/devices/pci0000:00/0000:00:1c.2/0000:06:00.0/net$ ls -l
total 0
drwxr-xr-x 5 root root 0 Aug 19 18:23 eth0

So the linking is there.

   Andrew

Re: DSA driver - how to glue to a PCI based NIC's mdio?

[Tim Harvey]

On Wed, Sep 30, 2015 at 2:40 PM, Andrew Lunn <andrew@lunn.ch> wrote:
>> > information to the NIC's device driver. Better would be to have a
>> > small shim driver which is loaded on your PCI_ID/DEVICE_ID. That would
>> > instantiate the NIC driver, and insert a DSA platform device.
>>
>> I was thinking of this as well, but then I would still need that shim
>> to know the netdevice that the driver I'm shimming creates so I can't
>> figure a way to do it without touching the PCI driver.
>
> You can probably get that from the PCI information you have:
>
> # /sys/devices/pci0000:00/0000:00:1c.2/0000:06:00.0 is the PCIe device. And it has
>
> # /sys/devices/pci0000:00/0000:00:1c.2/0000:06:00.0/net$ ls -l
> total 0
> drwxr-xr-x 5 root root 0 Aug 19 18:23 eth0
>
> So the linking is there.
>
>    Andrew

Andrew,

I read over the thread in April (https://lkml.org/lkml/2015/4/21/195)
regarding a USB Ethernet device with a switch attached as it discusses
a similar issue to mine where you do not have device-tree handles to
the mii bus or netdev and the device is enumerated. It does not seem
that any resolution was found there.

Do you know of any example of a 'shim' for a PCI device driver? I was
thinking you were talking about a PCI fixup or something that operated
on the device at enumeration time but in order to get hold of the
associated netdev I would need to instantiate the original NIC drive
then somehow get its mii_bus and netdev after its probe completed. I'm
not clear how to do that. How can I from my own PCI device driver, get
another PCI device driver's probe to fire, and following that, how
could I find that devices netdev and mdio_bus structures?

Another possibility discussed in that thread, that I have thought of
as well is doing this through a phylib driver. My device also has some
phy's hanging off two of the MV88E6xxx switch ports that need some
configuration so I 'already' have a phylib driver. However I haven't
figured out how to get hold of the netdev in order to register a DSA
platform device from there. The phylib probe function passes a
phy_device struct which gives me the mdio bus, but at that point the
attached_dev is NULL and remains that way until the NIC is brought up.
Currently I have a workqueue firing at 1Hz that checks for
fiber/copper/SFP state changes and in there I can check to see if the
phy_device has an attached_dev and when it does I can register a DSA
platform device and that works with the caviot that you have to bring
up the NIC that is attached to the switch in order for DSA to probe.

Regards,

Tim