Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[si-wei liu]

On 7/9/2018 6:00 AM, Roman Kagan wrote:
> On Tue, Jul 03, 2018 at 03:27:23PM -0700, si-wei liu wrote:
>> On 7/3/2018 2:58 AM, Roman Kagan wrote:
>>> So how is this coordination going to work?  One possibility is that the
>>> PV device emits a QMP event upon the guest driver confirming the support
>>> for failover, the management layer intercepts the event and performs
>>> device_add of the PT device.  Another is that the PT device is added
>>> from the very beginning (e.g. on the QEMU command line) but its parent
>>> PCI bridge subscribes a callback with the PV device to "activate" the PT
>>> device upon negotiating the failover feature.
>>>
>>> I think this needs to be decided within the scope of this patchset.
>> As what had been discussed in previous thread below, we would go with the
>> approach that QEMU manages the visibility of the PT device automatically.
>> Management layer supplies PT device to QEMU from the very beginning. This PT
>> device won't be exposed to guest immediately, unless or until the guest
>> virtio driver acknowledges the backup feature already. Once virtio driver in
>> the guest initiates a device reset, the corresponding PT device must be
>> taken out from guest. Then add it back later on after guest virtio completes
>> negotiation for the backup feature.
> This means that the parent bridge of the PT device (or whatever else can
> control the visibility of the PT device to the guest) will need to
> cooperate with the PV device *within* QEMU.  The most natural way to
> specify this connection is to have a property of one device to refer to
> the other by device-id.
This scheme has the problem that one device has to depend on the 
presence of the other - QEMU has the implication of the enumeration 
order if the two are not placed in the same birdge or PCI hierarchy. You 
can't get it reliably working if the bridge is going to be realized 
while the dependent PV device hadn't been yet, or vice versa.

> Another benefit of this approach is that it will allow to hide the
> (possibly transport-specific) device matching identifiers from the QEMU
> caller, as it won't need to be persistent nor visible to the management
> layer.  In particular, this will allow to move forward with the
> implementation of this PT-PV cooperation while the discussion of the
> matching scheme is still ongoing, because matching by MAC will certainly
> work as a first approximation.
The plan is to enable group ID based matching in the first place rather 
than match by MAC, the latter of which is fragile and problematic. I 
have made the Linux side changes and will get it posted once the QEMU 
discussion for grouping is finalized.

>
>>>>> Is the guest supposed to signal the datapath switch to the host?
>>>> No, guest doesn't need to be initiating datapath switch at all.
>>> What happens if the guest supports failover in its PV driver, but lacks
>>> the driver for the PT device?
>> The assumption of failover driver is that the primary (PT device) will be
>> able to get a datapath once it shows up in the guest .
> I wonder how universal this assumption is, given the variety of possible
> network configurations, including filters, VLANs, etc.  For whatever
> reason Hyper-V defines a control message over the PV device from guest
> to host for that.
These scenarios are different than the case with no driver support at 
all within guest. I think I particularly raised this as part of doing 
proper error handling when reviewing the original virtio failover patch 
- if failover module fails to enslave the VF due to guest network 
configurations, it has to signal virtio-net to propagate the error back 
to host. One way to handle that is to have virtio-net kick out a device 
reset and clear the feature bit upon re-negotiation, such that VF will 
be plugged out and won't get plugged in. I don't know for what reason 
the patch submitter did not incorporate that change. But it's in our 
plan to enhance that part, no worries.

>> If adding a PT device
>> to an unsupported guest, the result will be same as that without a standby
>> PV driver - basically got no networking as you don't get a working driver.
>>
>> Then perhaps don't add the PT device in the first place if guest lacks
>> driver support?
> You don't know this in advance.
 From migration point of view, it does not matter if guest lacks driver 
support for VF. I like to avoid duplicating hyper-v concept if at all 
possible. What makes sense with Hyper-V's accelerated networking doesn't 
have to work with KVM/QEMU SR-IOV live migration. Are you sure that the 
Hyper-V control message was added for this sole purpose? Seems to me an 
overkill for such an edge scenario.

>
>>>> However, QMP
>>>> events may be generated when exposing or hiding the PT device through hot
>>>> plug/unplug to facilitate host to switch datapath.
>>> The PT device hot plug/unplug are initiated by the host, aren't they?  Why
>>> would it also need QMP events for them?
>> As indicated above, the hot plug/unplug are initiated by QEMU not the
>> management layer. Hence the QMP hot plug event is used as an indicator to
>> switch host datapath. Unlike Windows Hyper-V SR-IOV driver model, the Linux
>> host network stack does not offer a fine grained PF driver API to move
>> MAC/VLAN filter, and the VF driver has to start with some initial MAC
>> address filter programmed in when present in the guest. The QMP event is
>> served as a checkpoint to switch MAC filter and/or VLAN filter between the
>> PV and the VF.
>>
> I'd appreciate something like a sequence diagram to better understand
> the whole picture...
>
>>>>> Is the scheme going to be applied/extended to other transports (vmbus,
>>>>> virtio-ccw, etc.)?
>>>> Well, it depends on the use case, and how feasible it can be extended to
>>>> other transport due to constraints and transport specifics.
>>>>
>>>>> Is the failover group concept going to be used beyond PT-PV network
>>>>> device failover?
>>>> Although the concept of failover group is generic, the implementation itself
>>>> may vary.
>>> My point with these two questions is that since this patchset is
>>> defining external interfaces -- with guest OS, with management layer --
>>> which are not easy to change later, it might make sense to try and see
>>> if the interfaces map to other usecases.  E.g. I think we can get enough
>>> information on how Hyper-V handles PT-PV network device failover from
>>> the current Linux implementation; it may be a good idea to share some
>>> concepts and workflows with virtio-pci.
>> As you may see from above, the handshake of virtio failover depends on hot
>> plug (PCI or ACPI) and virtio specifics (feature negotiation). So far as I
>> see the Hyper-V uses a completely different handshake protocol of its own
>> (guest initiated datapath switch, Serial number in VMBus PCI bridge) than
>> that of virtio. I can barely imagine how code could be implemented in a
>> shared manner, although I agree conceptually failover group between these
>> two is similar or the same.
> I actually think there must be a lot in common: the way for the
> management layer to specify the binding between the PT and PV devices;
> the overall sequence of state transitions of every component, the QMP
> events and the points in time when they are emitted, the way to adjust
> host-side network configuration and the time when to do it, and so on.
> It's unfortunate that the implementation of PV-PT failover in guest
> Linux happens to have diverged between virtio and hyperv, but I don't
> see any fundamental difference and I wouldn't be surprised if they
> eventually converged sooner rather than later.
(loop in Intel folks and Linux netdev)

Actually it's not without reason that Linux/virtio has to diverge from 
Hyper-V. The Hyper-V SR-IOV driver model allows VF be plugged in and 
registered with the stack without a MAC address filter programmed in the 
NIC, while Windows Hyper-V at the host side is able to move a MAC filter 
around from the NIC PV backend to VF upon receiving the DATAPATH_SWITCH 
control message initiated by guest. Windows NDIS has 
OID_RECEIVE_FILTER_MOVE_FILTER API to have PF update the MAC filter for 
the VF without involving guest or VF. For all of these there are no 
equivalent in the Linux SR-IOV driver model. How do you propose to have 
guest initiate the datapath switching at any point in time when you're 
dealing with Linux host network stack?

One may say we can plug in a VF with a random MAC filter programmed in 
prior, and initially use that random MAC within guest. This would require:

a) not relying on permanent MAC address to do pairing during the initial 
discovery, e.g. use the failover group ID as in this discussion
b) host to toggle the MAC address filter: which includes taking down the 
tap device to return the MAC back to PF, followed by assigning that MAC 
to VF using "ip link ... set vf ..."
c) notify guest to reload/reset VF driver for the change of hardware MAC 
address
d) until VF reloads the driver it won't be able to use the datapath, so 
very short period of network outage is (still) expected

But as you see this still diverges from the Hyper-V model. What do we 
buy for using a random address during initial discovery and requiring VF 
to complete the handshake? Less network downtime during datapath 
switching? Sorry but that's not a key factor at all for our main goal - 
live migration.

Regards,
-Siwei

>
> There are a few things that need to be specific for PT and/or PV
> transport, the matching identifier among them, but I guess a lot can
> still be in common.
>
> Roman.
>

Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Michael S. Tsirkin]

On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:
> The plan is to enable group ID based matching in the first place rather than
> match by MAC, the latter of which is fragile and problematic.

It isn't all that fragile - hyperv used same for a while, so if someone
posts working patches with QEMU support but before this grouping stuff,
I'll happily apply them.

> I have made
> the Linux side changes and will get it posted once the QEMU discussion for
> grouping is finalized.

Go ahead but at this point I think we need to see actual QEMU support at
least for the basic functionality before we merge more code Linux-side.

-- 
MST

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Siwei Liu]

On Mon, Jul 9, 2018 at 6:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:
>> The plan is to enable group ID based matching in the first place rather than
>> match by MAC, the latter of which is fragile and problematic.
>
> It isn't all that fragile - hyperv used same for a while, so if someone
> posts working patches with QEMU support but before this grouping stuff,
> I'll happily apply them.

I wouldn't box the solution to very limited scenario just because of
matching by MAC, the benefit of having generic group ID in the first
place is that we save the effort of maintaining legacy MAC based
pairing that just adds complexity anyway. Currently the VF's MAC
address cannot be changed by either PF or by the guest user is a
severe limitation due to this. The other use case is that PT device
than VF would generally have different MAC than the standby virtio. We
shouldn't limit itself to VF specific scenario from the very
beginning.

It's just small piece of QEMU code to add in, why not?

>
>> I have made
>> the Linux side changes and will get it posted once the QEMU discussion for
>> grouping is finalized.
>
> Go ahead but at this point I think we need to see actual QEMU support at
> least for the basic functionality before we merge more code Linux-side.

The VFIO visibility work is being worked on. We'll post patches when it's ready.

Thanks,
-Siwei


>
> --
> MST
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Cornelia Huck]

On Tue, 10 Jul 2018 17:07:37 -0700
Siwei Liu <loseweigh@gmail.com> wrote:

> On Mon, Jul 9, 2018 at 6:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:  
> >> The plan is to enable group ID based matching in the first place rather than
> >> match by MAC, the latter of which is fragile and problematic.  
> >
> > It isn't all that fragile - hyperv used same for a while, so if someone
> > posts working patches with QEMU support but before this grouping stuff,
> > I'll happily apply them.  
> 
> I wouldn't box the solution to very limited scenario just because of
> matching by MAC, the benefit of having generic group ID in the first
> place is that we save the effort of maintaining legacy MAC based
> pairing that just adds complexity anyway. Currently the VF's MAC
> address cannot be changed by either PF or by the guest user is a
> severe limitation due to this. The other use case is that PT device
> than VF would generally have different MAC than the standby virtio. We
> shouldn't limit itself to VF specific scenario from the very
> beginning.

So, this brings me to a different concern: the semantics of
VIRTIO_NET_F_STANDBY.

* The currently sole user seems to be the virtio-net Linux driver.
* The commit messages, code comments and Documentation/ all talk about
  matching by MAC.
* I could not find any proposed update to the virtio spec. (If there
  had been an older proposal with a different feature name, it is not
  discoverable.)

VIRTIO_NET_F_STANDBY is a host <-> guest interface. As there's no
official spec, you can only go by the Linux implementation, and by that
its semantics seem to be 'match by MAC', not 'match by other criteria'.

How is this supposed to work in the long run?

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Siwei Liu]

On Wed, Jul 11, 2018 at 2:53 AM, Cornelia Huck <cohuck@redhat.com> wrote:
> On Tue, 10 Jul 2018 17:07:37 -0700
> Siwei Liu <loseweigh@gmail.com> wrote:
>
>> On Mon, Jul 9, 2018 at 6:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> > On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:
>> >> The plan is to enable group ID based matching in the first place rather than
>> >> match by MAC, the latter of which is fragile and problematic.
>> >
>> > It isn't all that fragile - hyperv used same for a while, so if someone
>> > posts working patches with QEMU support but before this grouping stuff,
>> > I'll happily apply them.
>>
>> I wouldn't box the solution to very limited scenario just because of
>> matching by MAC, the benefit of having generic group ID in the first
>> place is that we save the effort of maintaining legacy MAC based
>> pairing that just adds complexity anyway. Currently the VF's MAC
>> address cannot be changed by either PF or by the guest user is a
>> severe limitation due to this. The other use case is that PT device
>> than VF would generally have different MAC than the standby virtio. We
>> shouldn't limit itself to VF specific scenario from the very
>> beginning.
>
> So, this brings me to a different concern: the semantics of
> VIRTIO_NET_F_STANDBY.
>
> * The currently sole user seems to be the virtio-net Linux driver.
> * The commit messages, code comments and Documentation/ all talk about
>   matching by MAC.
> * I could not find any proposed update to the virtio spec. (If there
>   had been an older proposal with a different feature name, it is not
>   discoverable.)

No, there was no such spec patch at all when the Linux patch was
submitted, hence match by MAC is the only means to pair device due to
lack of QEMU support.

Q: Does it work?
A: Well, it works for some.
Q: Does it work well to support all scenarios?
A: No, not as it claims to.
Q: Can it do better job to support all scenarios?
A: Yes, do pairing with the failover group ID instead.
Q: Does pairing still need to be MAC based if using failover group ID?
A: It depends, it's up to the implementation to verify MAC address
depending on the need (e.g. VF failover versus PT device replacement),
though MAC matching is no longer positioned as a requirement for
pairing or grouping.

There's no such stickiness for matching by MAC defined anywhere. The
semantics of VIRTIO_NET_F_STANDBY feature are mostly a failover
concept that the standby device should be used when the primary is not
present. We now have added the group ID on QEMU. I don't see why
bothering to get rid of the limitation: it's never been exposed. No
existing users. No API/ABI defined at all.

>
> VIRTIO_NET_F_STANDBY is a host <-> guest interface. As there's no
> official spec, you can only go by the Linux implementation, and by that
> its semantics seem to be 'match by MAC', not 'match by other criteria'.
>
> How is this supposed to work in the long run?

That group ID thing should work for all OS. Not just Linux.

I will change all the references to MAC matching in my upcoming patch.
Thank you for the note.

-Siwei

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Cornelia Huck]

On Thu, 12 Jul 2018 02:37:03 -0700
Siwei Liu <loseweigh@gmail.com> wrote:

> On Wed, Jul 11, 2018 at 2:53 AM, Cornelia Huck <cohuck@redhat.com> wrote:
> > On Tue, 10 Jul 2018 17:07:37 -0700
> > Siwei Liu <loseweigh@gmail.com> wrote:
> >  
> >> On Mon, Jul 9, 2018 at 6:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote:  
> >> > On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:  
> >> >> The plan is to enable group ID based matching in the first place rather than
> >> >> match by MAC, the latter of which is fragile and problematic.  
> >> >
> >> > It isn't all that fragile - hyperv used same for a while, so if someone
> >> > posts working patches with QEMU support but before this grouping stuff,
> >> > I'll happily apply them.  
> >>
> >> I wouldn't box the solution to very limited scenario just because of
> >> matching by MAC, the benefit of having generic group ID in the first
> >> place is that we save the effort of maintaining legacy MAC based
> >> pairing that just adds complexity anyway. Currently the VF's MAC
> >> address cannot be changed by either PF or by the guest user is a
> >> severe limitation due to this. The other use case is that PT device
> >> than VF would generally have different MAC than the standby virtio. We
> >> shouldn't limit itself to VF specific scenario from the very
> >> beginning.  
> >
> > So, this brings me to a different concern: the semantics of
> > VIRTIO_NET_F_STANDBY.
> >
> > * The currently sole user seems to be the virtio-net Linux driver.
> > * The commit messages, code comments and Documentation/ all talk about
> >   matching by MAC.
> > * I could not find any proposed update to the virtio spec. (If there
> >   had been an older proposal with a different feature name, it is not
> >   discoverable.)  
> 
> No, there was no such spec patch at all when the Linux patch was
> submitted, hence match by MAC is the only means to pair device due to
> lack of QEMU support.

We need to know what the device offers if it offers the feature bit,
and what it is supposed to do when the driver negotiates it. Currently,
we can only go by what the Linux driver does and what it expects. IOW,
we need a spec update proposal. Obviously, this should be discussed in
conjunction with the rest.

> 
> Q: Does it work?
> A: Well, it works for some.
> Q: Does it work well to support all scenarios?
> A: No, not as it claims to.
> Q: Can it do better job to support all scenarios?
> A: Yes, do pairing with the failover group ID instead.
> Q: Does pairing still need to be MAC based if using failover group ID?
> A: It depends, it's up to the implementation to verify MAC address
> depending on the need (e.g. VF failover versus PT device replacement),
> though MAC matching is no longer positioned as a requirement for
> pairing or grouping.

Whether matching by MAC is good or sufficient is a different
discussion. It is, however, what the code currently *does*, and in
absence of a spec update, it is the only reference for this feature.

> 
> There's no such stickiness for matching by MAC defined anywhere. The
> semantics of VIRTIO_NET_F_STANDBY feature are mostly a failover
> concept that the standby device should be used when the primary is not
> present. We now have added the group ID on QEMU. I don't see why
> bothering to get rid of the limitation: it's never been exposed. No
> existing users. No API/ABI defined at all.

This is scheduled to be released with the next Linux version, which is
right now in the -rc phase. It *is* API (a guest <-> host API).

No corresponding code is present in QEMU 3.0, which is in freeze right
now. Anything that goes into QEMU 3.1 or later needs to accommodate
Linux 4.18 as a guest.

> 
> >
> > VIRTIO_NET_F_STANDBY is a host <-> guest interface. As there's no
> > official spec, you can only go by the Linux implementation, and by that
> > its semantics seem to be 'match by MAC', not 'match by other criteria'.
> >
> > How is this supposed to work in the long run?  
> 
> That group ID thing should work for all OS. Not just Linux.

That's exactly my point: We need to care about not-Linux. And about
not-QEMU as well. A virtio feature bit should not be defined by what
Linux and QEMU do, but needs a real spec.

So, currently we have a Linux driver implementation that matches by
MAC. If a Linux version with this is released, every device that offers
VIRTIO_NET_F_STANDBY needs to support matching by MAC so that this
Linux driver will not break. Adding further matching methods should be
fine, but might need additional features (needs to be discussed).

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Siwei Liu]

_

On Thu, Jul 12, 2018 at 4:31 AM, Cornelia Huck <cohuck@redhat.com> wrote:
> On Thu, 12 Jul 2018 02:37:03 -0700
> Siwei Liu <loseweigh@gmail.com> wrote:
>
>> On Wed, Jul 11, 2018 at 2:53 AM, Cornelia Huck <cohuck@redhat.com> wrote:
>> > On Tue, 10 Jul 2018 17:07:37 -0700
>> > Siwei Liu <loseweigh@gmail.com> wrote:
>> >
>> >> On Mon, Jul 9, 2018 at 6:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> >> > On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:
>> >> >> The plan is to enable group ID based matching in the first place rather than
>> >> >> match by MAC, the latter of which is fragile and problematic.
>> >> >
>> >> > It isn't all that fragile - hyperv used same for a while, so if someone
>> >> > posts working patches with QEMU support but before this grouping stuff,
>> >> > I'll happily apply them.
>> >>
>> >> I wouldn't box the solution to very limited scenario just because of
>> >> matching by MAC, the benefit of having generic group ID in the first
>> >> place is that we save the effort of maintaining legacy MAC based
>> >> pairing that just adds complexity anyway. Currently the VF's MAC
>> >> address cannot be changed by either PF or by the guest user is a
>> >> severe limitation due to this. The other use case is that PT device
>> >> than VF would generally have different MAC than the standby virtio. We
>> >> shouldn't limit itself to VF specific scenario from the very
>> >> beginning.
>> >
>> > So, this brings me to a different concern: the semantics of
>> > VIRTIO_NET_F_STANDBY.
>> >
>> > * The currently sole user seems to be the virtio-net Linux driver.
>> > * The commit messages, code comments and Documentation/ all talk about
>> >   matching by MAC.
>> > * I could not find any proposed update to the virtio spec. (If there
>> >   had been an older proposal with a different feature name, it is not
>> >   discoverable.)
>>
>> No, there was no such spec patch at all when the Linux patch was
>> submitted, hence match by MAC is the only means to pair device due to
>> lack of QEMU support.
>
> We need to know what the device offers if it offers the feature bit,
> and what it is supposed to do when the driver negotiates it. Currently,
> we can only go by what the Linux driver does and what it expects. IOW,
> we need a spec update proposal. Obviously, this should be discussed in
> conjunction with the rest.

The definition is incomplete due to lack of spec. There's no "host"
part defined yet in the host-guest interface. If match by MAC is an
interface, the same must be done on the host(device) side as well,
which has been agreed not the way to go. However, I don't think that's
what the author intends to do by interpreting his QEMU patch - it
missed the other parts as well, such as the feature negotiation and
how it interacts with the paired device.

What I said is that match by MAC is just a guest implementation that
one can change at any time. We now have the group ID on QEMU, why
still sticking to matching by MAC? It shoulnd't be a host-guest
interface in the first place anyway.

>
>>
>> Q: Does it work?
>> A: Well, it works for some.
>> Q: Does it work well to support all scenarios?
>> A: No, not as it claims to.
>> Q: Can it do better job to support all scenarios?
>> A: Yes, do pairing with the failover group ID instead.
>> Q: Does pairing still need to be MAC based if using failover group ID?
>> A: It depends, it's up to the implementation to verify MAC address
>> depending on the need (e.g. VF failover versus PT device replacement),
>> though MAC matching is no longer positioned as a requirement for
>> pairing or grouping.
>
> Whether matching by MAC is good or sufficient is a different
> discussion. It is, however, what the code currently *does*, and in
> absence of a spec update, it is the only reference for this feature.

Not really, it's the same discussion. Before the group ID discussion I
explicitly asked for the spec for VIRTIO_NET_F_STANDBY about the
semantics.  Since no one come up with a spec, I assumed it is still
being worked on and it was all right to have the initial Linux
implementation to be in. I assume that is a formal process while
working on a complicated feature the spec can come later once
everything becomes clear along with the discussions.

https://www.spinics.net/lists/netdev/msg499011.html

I made the same claim at the time that we shouldn't go with what's
been implemented in Linux, but instead should look at the entire
picture to decide what should be the right semantics for
VIRTIO_NET_F_STANDBY. I agree with you we need a spec. I can work on a
spec but I'd like to clarify that there was nothing defined yet for
VIRTIO_NET_F_STANDBY so it shouldn't be called as spec update. It's
still a work-in-progress feature that IMHO it's different than the
situation that there used to be pre-spec virtio implementation already
working on both host and guest side. The current VIRTIO_NET_F_STANDBY
implementation in Linux is pretty much dead code without a spec
followed by a host/device side implementation.

-Siwei


>
>>
>> There's no such stickiness for matching by MAC defined anywhere. The
>> semantics of VIRTIO_NET_F_STANDBY feature are mostly a failover
>> concept that the standby device should be used when the primary is not
>> present. We now have added the group ID on QEMU. I don't see why
>> bothering to get rid of the limitation: it's never been exposed. No
>> existing users. No API/ABI defined at all.
>
> This is scheduled to be released with the next Linux version, which is
> right now in the -rc phase. It *is* API (a guest <-> host API).
>
> No corresponding code is present in QEMU 3.0, which is in freeze right
> now. Anything that goes into QEMU 3.1 or later needs to accommodate
> Linux 4.18 as a guest.
>
>>
>> >
>> > VIRTIO_NET_F_STANDBY is a host <-> guest interface. As there's no
>> > official spec, you can only go by the Linux implementation, and by that
>> > its semantics seem to be 'match by MAC', not 'match by other criteria'.
>> >
>> > How is this supposed to work in the long run?
>>
>> That group ID thing should work for all OS. Not just Linux.
>
> That's exactly my point: We need to care about not-Linux. And about
> not-QEMU as well. A virtio feature bit should not be defined by what
> Linux and QEMU do, but needs a real spec.
>
> So, currently we have a Linux driver implementation that matches by
> MAC. If a Linux version with this is released, every device that offers
> VIRTIO_NET_F_STANDBY needs to support matching by MAC so that this
> Linux driver will not break. Adding further matching methods should be
> fine, but might need additional features (needs to be discussed).

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Michael S. Tsirkin]

On Thu, Jul 12, 2018 at 01:52:53PM -0700, Siwei Liu wrote:
> The definition is incomplete due to lack of spec. There's no "host"
> part defined yet in the host-guest interface. If match by MAC is an
> interface, the same must be done on the host(device) side as well,
> which has been agreed not the way to go. However, I don't think that's
> what the author intends to do by interpreting his QEMU patch - it
> missed the other parts as well, such as the feature negotiation and
> how it interacts with the paired device.
> 
> What I said is that match by MAC is just a guest implementation that
> one can change at any time. We now have the group ID on QEMU, why
> still sticking to matching by MAC? It shoulnd't be a host-guest
> interface in the first place anyway.

I think that match by MAC is a simple portable way to match devices.
E.g. it will work seamlessly with niche things like zPCI. However
there are other niche use-cases that aren't addressed by match by MAC
such as PF pass-through as a primary, and the pci bridge trick addresses
that at cost of some portability.

So I see no issues supporting both mechanisms, but others on the TC
might feel differently.

-- 
MST

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Siwei Liu]

On Thu, Jul 12, 2018 at 2:00 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Thu, Jul 12, 2018 at 01:52:53PM -0700, Siwei Liu wrote:
>> The definition is incomplete due to lack of spec. There's no "host"
>> part defined yet in the host-guest interface. If match by MAC is an
>> interface, the same must be done on the host(device) side as well,
>> which has been agreed not the way to go. However, I don't think that's
>> what the author intends to do by interpreting his QEMU patch - it
>> missed the other parts as well, such as the feature negotiation and
>> how it interacts with the paired device.
>>
>> What I said is that match by MAC is just a guest implementation that
>> one can change at any time. We now have the group ID on QEMU, why
>> still sticking to matching by MAC? It shoulnd't be a host-guest
>> interface in the first place anyway.
>
> I think that match by MAC is a simple portable way to match devices.
> E.g. it will work seamlessly with niche things like zPCI. However

That's a good point. I'm not sure if it's a valid assumption that zPCI
should always use the same MAC address as that of virtio. Someone
who's more familiar with the use case may decide and work on that. It
means VFIO device has to take in the MAC address as an identifier to
the "-device vfio-pci,.." QEMU option. I think there's no point to
match device using group ID in QEMU while using MAC in the guest.
Based on that assumption, I'd go with making VIRTIO_NET_F_STANDBY to
match device based on group ID, while someone may come up with another
feature bit later, say VIRTIO_NET_F_STANDBY_BY_MAC when its QEMU
support is available. Would it make sense?

-Siwei

> there are other niche use-cases that aren't addressed by match by MAC
> such as PF pass-through as a primary, and the pci bridge trick addresses
> that at cost of some portability.
>
> So I see no issues supporting both mechanisms, but others on the TC
> might feel differently.
>
> --
> MST

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Samudrala, Sridhar]

On 7/12/2018 6:19 PM, Siwei Liu wrote:
> On Thu, Jul 12, 2018 at 2:00 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
>> On Thu, Jul 12, 2018 at 01:52:53PM -0700, Siwei Liu wrote:
>>> The definition is incomplete due to lack of spec. There's no "host"
>>> part defined yet in the host-guest interface. If match by MAC is an
>>> interface, the same must be done on the host(device) side as well,
>>> which has been agreed not the way to go. However, I don't think that's
>>> what the author intends to do by interpreting his QEMU patch - it
>>> missed the other parts as well, such as the feature negotiation and
>>> how it interacts with the paired device.
>>>
>>> What I said is that match by MAC is just a guest implementation that
>>> one can change at any time. We now have the group ID on QEMU, why
>>> still sticking to matching by MAC? It shoulnd't be a host-guest
>>> interface in the first place anyway.
>> I think that match by MAC is a simple portable way to match devices.
>> E.g. it will work seamlessly with niche things like zPCI. However
> That's a good point. I'm not sure if it's a valid assumption that zPCI
> should always use the same MAC address as that of virtio. Someone
> who's more familiar with the use case may decide and work on that. It
> means VFIO device has to take in the MAC address as an identifier to
> the "-device vfio-pci,.." QEMU option. I think there's no point to
> match device using group ID in QEMU while using MAC in the guest.
> Based on that assumption, I'd go with making VIRTIO_NET_F_STANDBY to
> match device based on group ID, while someone may come up with another
> feature bit later, say VIRTIO_NET_F_STANDBY_BY_MAC when its QEMU
> support is available. Would it make sense?

VIRTIO_NET_F_STANDBY as defined in the guest virtio_net driver supports match
by MAC address. I think we should add support for this feature bit in QEMU.
If submitting a patch to update the spec is a pre-requisite to add this
feature bit to QEMU, i can do that.

As far as i understand, group id patches to QEMU are still under review.
Matching by group ID can be another feature bit that could support matching
by group id as well as MAC.


> -Siwei
>
>> there are other niche use-cases that aren't addressed by match by MAC
>> such as PF pass-through as a primary, and the pci bridge trick addresses
>> that at cost of some portability.
>>
>> So I see no issues supporting both mechanisms, but others on the TC
>> might feel differently.
>>
>> --
>> MST

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Michael S. Tsirkin]

On Thu, Jul 12, 2018 at 09:20:41PM -0400, Samudrala, Sridhar wrote:
> On 7/12/2018 6:19 PM, Siwei Liu wrote:
> > On Thu, Jul 12, 2018 at 2:00 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > On Thu, Jul 12, 2018 at 01:52:53PM -0700, Siwei Liu wrote:
> > > > The definition is incomplete due to lack of spec. There's no "host"
> > > > part defined yet in the host-guest interface. If match by MAC is an
> > > > interface, the same must be done on the host(device) side as well,
> > > > which has been agreed not the way to go. However, I don't think that's
> > > > what the author intends to do by interpreting his QEMU patch - it
> > > > missed the other parts as well, such as the feature negotiation and
> > > > how it interacts with the paired device.
> > > > 
> > > > What I said is that match by MAC is just a guest implementation that
> > > > one can change at any time. We now have the group ID on QEMU, why
> > > > still sticking to matching by MAC? It shoulnd't be a host-guest
> > > > interface in the first place anyway.
> > > I think that match by MAC is a simple portable way to match devices.
> > > E.g. it will work seamlessly with niche things like zPCI. However
> > That's a good point. I'm not sure if it's a valid assumption that zPCI
> > should always use the same MAC address as that of virtio. Someone
> > who's more familiar with the use case may decide and work on that. It
> > means VFIO device has to take in the MAC address as an identifier to
> > the "-device vfio-pci,.." QEMU option. I think there's no point to
> > match device using group ID in QEMU while using MAC in the guest.
> > Based on that assumption, I'd go with making VIRTIO_NET_F_STANDBY to
> > match device based on group ID, while someone may come up with another
> > feature bit later, say VIRTIO_NET_F_STANDBY_BY_MAC when its QEMU
> > support is available. Would it make sense?
> 
> VIRTIO_NET_F_STANDBY as defined in the guest virtio_net driver supports match
> by MAC address. I think we should add support for this feature bit in QEMU.
> If submitting a patch to update the spec is a pre-requisite to add this
> feature bit to QEMU, i can do that.

It's not strictly a prerequisite but we need it in spec all the same, so
pls do that.

> As far as i understand, group id patches to QEMU are still under review.
> Matching by group ID can be another feature bit that could support matching
> by group id as well as MAC.
> 
> 
> > -Siwei
> > 
> > > there are other niche use-cases that aren't addressed by match by MAC
> > > such as PF pass-through as a primary, and the pci bridge trick addresses
> > > that at cost of some portability.
> > > 
> > > So I see no issues supporting both mechanisms, but others on the TC
> > > might feel differently.
> > > 
> > > --
> > > MST

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Cornelia Huck]

On Thu, 12 Jul 2018 21:20:41 -0400
"Samudrala, Sridhar" <sridhar.samudrala@intel.com> wrote:

> On 7/12/2018 6:19 PM, Siwei Liu wrote:
> > On Thu, Jul 12, 2018 at 2:00 PM, Michael S. Tsirkin <mst@redhat.com> wrote:  
> >> On Thu, Jul 12, 2018 at 01:52:53PM -0700, Siwei Liu wrote:  
> >>> The definition is incomplete due to lack of spec. There's no "host"
> >>> part defined yet in the host-guest interface. If match by MAC is an
> >>> interface, the same must be done on the host(device) side as well,
> >>> which has been agreed not the way to go. However, I don't think that's
> >>> what the author intends to do by interpreting his QEMU patch - it
> >>> missed the other parts as well, such as the feature negotiation and
> >>> how it interacts with the paired device.
> >>>
> >>> What I said is that match by MAC is just a guest implementation that
> >>> one can change at any time. We now have the group ID on QEMU, why
> >>> still sticking to matching by MAC? It shoulnd't be a host-guest
> >>> interface in the first place anyway.  
> >> I think that match by MAC is a simple portable way to match devices.
> >> E.g. it will work seamlessly with niche things like zPCI. However  
> > That's a good point. I'm not sure if it's a valid assumption that zPCI
> > should always use the same MAC address as that of virtio. Someone

I think we can mostly disregard the weirdness that is zPCI right now.
There should be no fundamental reasons that matching by MAC would not
work, though.

> > who's more familiar with the use case may decide and work on that. It
> > means VFIO device has to take in the MAC address as an identifier to
> > the "-device vfio-pci,.." QEMU option. I think there's no point to
> > match device using group ID in QEMU while using MAC in the guest.
> > Based on that assumption, I'd go with making VIRTIO_NET_F_STANDBY to
> > match device based on group ID, while someone may come up with another
> > feature bit later, say VIRTIO_NET_F_STANDBY_BY_MAC when its QEMU
> > support is available. Would it make sense?  
> 
> VIRTIO_NET_F_STANDBY as defined in the guest virtio_net driver supports match
> by MAC address. I think we should add support for this feature bit in QEMU.
> If submitting a patch to update the spec is a pre-requisite to add this
> feature bit to QEMU, i can do that.

Doing a spec patch and implementing matching by MAC in QEMU sounds like
a good plan to me.

> 
> As far as i understand, group id patches to QEMU are still under review.
> Matching by group ID can be another feature bit that could support matching
> by group id as well as MAC.

That plan sounds good to me as well.

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Michael S. Tsirkin]

On Wed, Jul 11, 2018 at 11:53:44AM +0200, Cornelia Huck wrote:
> On Tue, 10 Jul 2018 17:07:37 -0700
> Siwei Liu <loseweigh@gmail.com> wrote:
> 
> > On Mon, Jul 9, 2018 at 6:54 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> > > On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:  
> > >> The plan is to enable group ID based matching in the first place rather than
> > >> match by MAC, the latter of which is fragile and problematic.  
> > >
> > > It isn't all that fragile - hyperv used same for a while, so if someone
> > > posts working patches with QEMU support but before this grouping stuff,
> > > I'll happily apply them.  
> > 
> > I wouldn't box the solution to very limited scenario just because of
> > matching by MAC, the benefit of having generic group ID in the first
> > place is that we save the effort of maintaining legacy MAC based
> > pairing that just adds complexity anyway. Currently the VF's MAC
> > address cannot be changed by either PF or by the guest user is a
> > severe limitation due to this. The other use case is that PT device
> > than VF would generally have different MAC than the standby virtio. We
> > shouldn't limit itself to VF specific scenario from the very
> > beginning.
> 
> So, this brings me to a different concern: the semantics of
> VIRTIO_NET_F_STANDBY.
> 
> * The currently sole user seems to be the virtio-net Linux driver.
> * The commit messages, code comments and Documentation/ all talk about
>   matching by MAC.
> * I could not find any proposed update to the virtio spec. (If there
>   had been an older proposal with a different feature name, it is not
>   discoverable.)
> 
> VIRTIO_NET_F_STANDBY is a host <-> guest interface. As there's no
> official spec, you can only go by the Linux implementation, and by that
> its semantics seem to be 'match by MAC', not 'match by other criteria'.
> 
> How is this supposed to work in the long run?

We definitely need a spec patch for VIRTIO_NET_F_STANDBY documenting existing
semantics.  Sridhar, do you plan to take a look?

-- 
MST

Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Michael S. Tsirkin]

On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:
> What do we buy
> for using a random address during initial discovery and requiring VF to
> complete the handshake?

I don't see advantages to using a random address that is then
changed either: changing a MAC causes network downtime for most users.

> Less network downtime during datapath switching?
> Sorry but that's not a key factor at all for our main goal - live migration.

Isn't avoiding downtime what makes the migration "live"?
If you don't care about it at all just remove the device
and migrate without all these tricks.


-- 
MST

Re: Re: [Qemu-devel] [PATCH v3 0/3] Use of unique identifier for pairing virtio and passthrough devices...

[Siwei Liu]

On Mon, Jul 9, 2018 at 6:58 PM, Michael S. Tsirkin <mst@redhat.com> wrote:
> On Mon, Jul 09, 2018 at 06:11:53PM -0700, si-wei liu wrote:
>> What do we buy
>> for using a random address during initial discovery and requiring VF to
>> complete the handshake?
>
> I don't see advantages to using a random address that is then
> changed either: changing a MAC causes network downtime for most users.
>
Definitely.

I see Linux host stack fundamentally different with Windows, it's
non-sense to duplicate what Hyper-V is doing especially if there's no
extra benefit.

>> Less network downtime during datapath switching?
>> Sorry but that's not a key factor at all for our main goal - live migration.
>
> Isn't avoiding downtime what makes the migration "live"?
> If you don't care about it at all just remove the device
> and migrate without all these tricks.

Apparently this downtime is not avoidable even if guest initiates the
switch-over when it is done on Linux host stack. Unless the NIC
supports adding duplicate MAC filters with one has higher priority
than the other when both are present. I feel there's very little or
perhaps zero improvement for the downtime if moving to a
guest-initiated datapath switching model.

However, since this downtime is intermittent and generally
unnoticeable with a few packet drops, network should be resilient in
recovering from the drops. My point is that unless we can move to a
datapath switching model with zero downtime theoretically, this kind
of minor optimization offers very little help in general.

Regards,
-Siwei



>
>
> --
> MST
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: virtio-dev-unsubscribe@lists.oasis-open.org
> For additional commands, e-mail: virtio-dev-help@lists.oasis-open.org
>