Re: [PATCH net-next 00/16] nfp: ctrl vNIC

[Jiri Pirko <jiri@resnulli.us>]

Tue, Jun 06, 2017 at 09:21:45AM CEST, kubakici@wp.pl wrote:
>On Tue, 6 Jun 2017 08:16:10 +0200, Jiri Pirko wrote:
>> Tue, Jun 06, 2017 at 02:01:41AM CEST, jakub.kicinski@netronome.com wrote:
>> >Hi!
>> >
>> >This series adds the ability to use one vNIC as a control channel
>> >for passing messages to and from the application firmware.  The
>> >implementation restructures the existing netdev vNIC code to be able
>> >to deal with nfp_nets with netdev pointer set to NULL.  Control vNICs
>> >are not visible to userspace (other than for dumping ring state), and
>> >since they don't have netdevs we use a tasklet for RX and simple skb 
>> >list for TX queuing.
>> >
>> >Due to special status of the control vNIC we have to reshuffle the
>> >init code a bit to make sure control vNIC will be fully brought up
>> >(and therefore communication with app FW can happen) before any netdev
>> >or port is visible to user space.
>> >
>> >FW will designate which vNIC is supposed to be used as control one
>> >by setting _pf%u_net_ctrl_bar symbol.  Some FWs depend on metadata
>> >being prepended to control message, some prefer to look at queue ID
>> >to decide that something is a control message.  Our implementation
>> >can cater to both.
>> >
>> >First two users of this code will be eBPF maps and flower offloads.  
>> 
>> How do you actually do the configuration from the userspace? I did not
>> find it in the patches.
>
>Yes, there is nothing interesting in those patches, really.  It's all
>internal to the driver.  This set basically allows us to dedicate some
>queue pairs to high-speed communication with the FW (sending commands,
>populating/dumping eBPF and Flower tables).
>
>The eBPF maps and Flower offload patches should follow in coming weeks.

Okay. So you don't let the user to dicide for which purpose (bpf/flowe)
he wants to use the nic, right?


>
>> I'm not really sure that doing it using one "control netdevice" is the
>> correct way to go. 
>
>The control queues are not exposed to user space at all.  It's up to
>the driver logic to generate and consume all messages passing on those
>queues.  I had to make this a separate series because of the amount
>of code churn.  I'm also not a fan of exposing control netdevices.
>Reasons ranging from NetworkManager latching onto them (even if it's
>configured not to) to someone inevitably trying to control the device
>from user space with vendor commands and make the driver-kept state go
>out of whack :/

Okay, that got me confused. I thought that you want the control
netdevice. Now that is clear we both don't want to see that, I'm fine :)


>
>> The configuration is asic-wide, should be done by a devlink parent
>> handle which was introduced for that exact purpose.
>>
>> Am I missing something? We need to sync in this. In mlxsw we need to do
>> some pre-netdev configuraton as well.
>
>For programmable NICs we still need to come up with some API for setting
>the target application/firmware name/adapter mode, but I don't think we
>have any more fine-grained parameters to set per-device just yet.
>
>I was thinking of either devlink, or perhaps, since I'm hearing people
>want those to be persistent/written into flash, the recent MTD
>discussion got me wondering if we should just expose the entire flash
>and teach ethtool to modify the vendor-specific parameter table
>directly in the flash.  That seems like something that could quickly
>get out of hand, though :S

Yeah, I'm not fan of the mtd exposing some fw internals. I believe that
for the config like yours (switching the operation mode of nic), we
should have well defined user api. And since this is per-device, not
per-netdev, devlink should be the vahicle to carry this.


>
>What were your plans with pre-netdev config?

We need to pass come initial resource division. Generally the consensus
is to have these options exposed through devlink, let the user configure
them all and then to have a trigger that would cause driver
re-orchestration according to the new values. The flow would look like
this:

-driver loads with defaults, inits hw and instantiates netdevs
-driver exposes config options via devlink
-user sets up the options
-user pushes the "go" trigger
-upon the trigger command, devlink calls the driver re-init callback
-driver shuts down the current instances, re-initializes hw,
 re-instantiates the netdevs

Makes sense?


>
>I hope this makes things slightly clearer, sorry for not doing a good
>job on the cover letter :)

No worries :)