Re: [RFC] current devlink extension plan for NICs

[Andy Gospodarek <andy@greyhouse.net>]

On Sat, Mar 21, 2020 at 10:35:25AM +0100, Jiri Pirko wrote:
> Fri, Mar 20, 2020 at 10:25:08PM CET, kuba@kernel.org wrote:
> >On Fri, 20 Mar 2020 08:35:55 +0100 Jiri Pirko wrote:
> >> Fri, Mar 20, 2020 at 04:32:53AM CET, kuba@kernel.org wrote:
> >> >On Thu, 19 Mar 2020 20:27:19 +0100 Jiri Pirko wrote:  
[...]
> >
> >Also, once the PFs are created user may want to use them together 
> >or delegate to a VM/namespace. So when I was thinking we'd need some 
> >sort of a secure handshake between PFs and FW for the host to prove 
> >to FW that the PFs belong to the same domain of control, and their
> >resources (and eswitches) can be pooled.
> >
> >I'm digressing..
> 
> Yeah. This needs to be sorted out.
> 
> 
> >
> >> Now the PF itself can have a "nested eswitch" to manage. The "parent
> >> eswitch" where the PF was created would only see one leg to the "nested
> >> eswitch".
> >> 
> >> This "nested eswitch management" might or might not be required. Depends
> >> on a usecare. The question was, how to configure that I as a user
> >> want this or not.
> >
> >Ack. I'm extending your question. I think the question is not only who
> >controls the eswitch but also which PFs share the eswitch.
> 
> Yes.
> 

So we have implemented the notion of an 'adminstrative PF.'  This is a
gross simplification, but the idea is that the PCI domain (or CPU
complex) that contains this PF is the one that is 'in-charge' of the
eSwitch and the rest of the resources (firmware/phycode update) and
might also be the one that gets the VF representors when VFs are created
on any other PCI host/domains.

I'm not sure we need a kernel API to set it as I would leave this as
something that might be burned into the hardware in some manner.

> >
> >I think eswitch is just one capability, but SmartNIC will want to
> >control which ports see what capabilities in general. crypto offloads
> >and such.
> >
> >I presume in your model if host controls eswitch the smartNIC sees just
> 
> host may control the "nested eswitch" in the SmartNIC case.
> 

I'm not sure programming the eswitch in a nested manner is realistic.
Sure we can make hardware do it, but it's probably more trouble than
it's worth.  If a smartnic wants to give control of flows to the host
then it makes more sense to allow some communication at a higher layer
so that requests for hardware offload can be easily validated against
some sort of policy set forth by the admin of the smartnic.

> >what what comes out of Hosts single "uplink"? What if SmartNIC wants
> >the host to be able to control the forwarding but not loose the ability
> >to tap the VF to VF traffic?
> 
> You mean that the VF representors would be in both SmartNIC host and
> host? I don't know how that could work. I think it has to be either
> there or there.
> 

Agreed.  The VF reps should probably appear on whichever host/domain has
the Admin PF.