Re: [RFC] current devlink extension plan for NICs

[Jason Gunthorpe <jgg@ziepe.ca>]

On Mon, Mar 23, 2020 at 08:56:19PM -0700, Jakub Kicinski wrote:
> On Mon, 23 Mar 2020 19:06:05 -0300 Jason Gunthorpe wrote:
> > On Mon, Mar 23, 2020 at 12:21:23PM -0700, Jakub Kicinski wrote:
> > > > >I see so you want the creation to be controlled by the same entity that
> > > > >controls the eswitch..
> > > > >
> > > > >To me the creation should be on the side that actually needs/will use
> > > > >the new port. And if it's not eswitch manager then eswitch manager
> > > > >needs to ack it.    
> > > > 
> > > > Hmm. The question is, is it worth to complicate things in this way?
> > > > I don't know. I see a lot of potential misunderstandings :/  
> > > 
> > > I'd see requesting SFs over devlink/sysfs as simplification, if
> > > anything.  
> > 
> > We looked at it for a while, working the communication such that the
> > 'untrusted' side could request a port be created with certain
> > parameters and the 'secure eswitch' could know those parameters to
> > authorize and wire it up was super complicated and very hard to do
> > without races.
> > 
> > Since it is a security sensitive operation it seems like a much more
> > secure design to have the secure side do all the creation and present
> > the fully operational object to the insecure side.
> > 
> > To draw a parallel to qemu & kvm, the untrusted guest VM can't request
> > that qemu create a virtio-net for it. Those are always hot plugged in
> > by the secure side. Same flow here.
> 
> Could you tell us a little more about the races? Other than the
> communication channel what changes between issuing from cloud API
> vs devlink?

If I recall the problems came when trying to work with existing cloud
infrastructure that doesn't assume this operating model. You need to
somehow adapt an async APIs of secure/insecure communication with an
async API inside the cloud world. It was a huge mess.

> Side note - there is no communication channel between VM and hypervisor
> right now, which is the cause for weird designs e.g. the failover/auto
> bond mechanism.

Right, and considering the security concerns building one hidden
inside a driver seems like a poor idea..

> > The VF model is poor because the VF is just a dummy stub until the
> > representor/eswitch side is fully configured. There is no way for the
> > Linux driver to know if the VF is operational or not, so we get weird
> > artifacts where we sometimes bind a driver to a VF (and get a
> > non-working ethXX) and sometimes we don't. 
> 
> Sounds like an implementation issue :S

How so?

> > The only reason it is like this is because of how SRIOV requires
> > everything to be preallocated.
> 
> SF also requires pre-allocated resources, so you're not talking about
> PCI mem space etc. here I assume.

It isn't pre-allocated, the usage of the BAR space is dynamic.

> > The SFs can't even exist until they are configured, so there is no
> > state where a driver is connected to an inoperative SF.
> 
> You mean it doesn't exist in terms of sysfs device entry?

I mean literally do not exist at the HW level.

Jason