Re: [PATCH net-next v2 4/7] devlink: allow subports on devlink PCI ports

Jiri Pirko
2019-03-13T16:33:13Z

Wed, Mar 13, 2019 at 05:17:31PM CET, jakub.kicinski@netronome.com wrote:
>On Wed, 13 Mar 2019 07:07:01 +0100, Jiri Pirko wrote:
>> Tue, Mar 12, 2019 at 09:56:28PM CET, jakub.kicinski@netronome.com wrote:
>> >On Tue, 12 Mar 2019 15:02:39 +0100, Jiri Pirko wrote:
>> >> Tue, Mar 12, 2019 at 03:10:54AM CET, wrote:
>> >> >On Mon, 11 Mar 2019 09:52:04 +0100, Jiri Pirko wrote:
>> >> >> Fri, Mar 08, 2019 at 08:09:43PM CET, wrote:
>> >> >> >If the switchport is in the hypervisor then only the hypervisor can
>> >> >> >control switching/forwarding, correct?
>> >> >>
>> >> >> Correct.
>> >> >>
>> >> >> >The primary use case for partitioning within a VM (of a VF) would be
>> >> >> >containers (and DPDK)?
>> >> >>
>> >> >> Makes sense.
>> >> >>
>> >> >> >SR-IOV makes things harder. Splitting a PF is reasonably easy to grasp.
>> >> >> >I'm trying to get a sense of is how would we control an SR-IOV
>> >> >> >environment as a whole.
>> >> >>
>> >> >> You mean orchestration?
>> >> >
>> >> >Right, orchestration.
>> >> >
>> >> >To be clear on where I'm going with this - if we want to allow VFs
>> >> >to partition themselves then they have to control what is effectively
>> >> >a "nested" switch. A per-VF set of rules which would the get
>> >>
>> >> Wait. If you allow to make VF subports (I believe that is what you ment
>> >> by VFs partition themselves), that does not mean they will have a
>> >> separate nested switch. They would still belong under the same one.
>> >
>> >But that existing switch is administered by the hypervisor, how would
>> >the VF owners install forwarding rules in a switch they don't control?
>>
>> They won't.
>
>Argh. So how is forwarding configured if there are no rules? Are you
>going to assume its switching on MACs? We're supposed to offload
>software constructs. If its a software port it needs to be explicitly
>switched. If it's not explicitly switched - we already have macvlan
>offload.

Wait a second. You configure the switch. And for that, you have the
switchports (representors). What we are talking about are VF (VF
subport) host legs. Am I missing something?

>
>> >> >"flattened" into the main eswitch rule set. If I was to choose I'd
>> >> >really rather have this "flattening" be done on the (Linux) hypervisor
>> >> >and not in the vendor driver and firmware.
>> >>
>> >> Agreed. Driver should provide one big switch. User should configure it.
>> >
>> >Cool, when you say user - is it the tenant or the provider?
>>
>> Whoever gets access to the instance.
>>
>> >> >I'd much rather have the VM make a "give me another NIC" orchestration
>> >> >call via some high level REST API than devlink. This makes the
>> >> >configuration strictly high level to low level:
>> >> >
>> >> > VM -> cloud net REST API -> cloud agent -> devlink/Linux -> FW -> HW
>> >> >
>> >> >Without round trips via firmware.
>> >>
>> >> Okay. So the "devlink/Linux -> FW" part is going to happen on baremetal.
>> >> Makes sense.
>> >>
>> >> >This allows for easy policy enforcement, common code to be maintained
>> >> >in user space, in high level languages (no 0.5M LoC drivers and 10M LoC
>> >> >firmware for every driver). It can also be used with software paths
>> >> >like VirtIO..
>> >>
>> >> Agreed.
>> >>
>> >> >Modelling and debugging a nested switch would be a nightmare. What
>> >> >follows is that we probably shouldn't deal with partitioning of VFs,
>> >> >but rather only partition via the PF devlink instance, and reassign
>> >> >the partitions to VMs.
>> >>
>> >> Agreed. That must be misunderstanding, I never suggested nested
>> >> switches.
>> >
>> >Cool, yes, I was making sure we weren't going in that direction :)
>>
>> Okay.
>>
>> >> >> I originally planned to implement sriov orchestration api in devlink too.
>> >> >
>> >> >Interesting, would you mind elaborating?
>> >>
>> >> I have to think about it. But something like this:
>> >> [...]
>> >
>> >I see thanks for the examples, they makes things clear!
>>
>> Okay. I will put together some documentation including this. I have some
>> patches that implement some of the stuff. Your patchset also does some
>> of that (considering you adjust a thing or two). Lets make this right.
>
>Yeah, I feel like I'm again getting further from clarity on what you're
>trying to achieve.

:)