From mboxrd@z Thu Jan 1 00:00:00 1970 From: Alex Williamson Subject: Re: [libvirt] Expose vfio device display/migration to libvirt and above, was Re: [PATCH 0/3] sample: vfio mdev display devices. Date: Thu, 10 May 2018 09:57:02 -0600 Message-ID: <20180510095702.4da6d6eb@w520.home> References: <20180424165918.5c2ef037@w520.home> <0a1d6487-0dfb-2ffc-4774-ebaf65c15892@nvidia.com> <20180425120057.0fabb70e@w520.home> <20180425195229.GK2496@work-vm> <20180426185522.GQ2631@work-vm> <20180503125800.76cc7582@w520.home> <20180504074944.GD8859@erzo-ntb> <20180504100344.221e399f@t450s.home> <20180510110029.GA9645@erzo-ntb> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Cc: Neo Jia , kvm@vger.kernel.org, libvirt , "Dr. David Alan Gilbert" , Tina Zhang , Jiri, Kirti Wankhede , Gerd Hoffmann , Laine Stump , Denemark , intel-gvt-dev@lists.freedesktop.org To: Erik Skultety Return-path: In-Reply-To: <20180510110029.GA9645@erzo-ntb> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: libvir-list-bounces@redhat.com Errors-To: libvir-list-bounces@redhat.com List-Id: kvm.vger.kernel.org On Thu, 10 May 2018 13:00:29 +0200 Erik Skultety wrote: > ... > > > > Now, if we (theoretically) can settle on easing the restrictions Alex > > > has mentioned, we in fact could introduce a QMP command to probe > > > these devices and provide libvirt with useful information at that > > > point in time. Of course, since the 3rd party vendor is "de-coupled" > > > from qemu, libvirt would have no way to find out that the driver has > > > changed in the meantime, thus still using the old information we > > > gathered, ergo potentially causing the QEMU process to fail > > > eventually. But then again, there's very often a strong > > > recommendation to reboot your host after a driver update, especially > > > in NVIDIA's case, which means this fact wouldn't matter. However, > > > there's also a significant drawback to my proposal which probably > > > renders it completely useless (but we can continue from there...) and > > > that is the devices would either have to be present already (not an > > > option) or QEMU would need to be enhanced in a way, that it would > > > create a dummy device during QMP probing, open it, collect the > > > information libvirt needs, close it and remove it. If the driver > > > doesn't change in the meantime, this should be sufficient for a VM to > > > be successfully instantiated with a display, right? > > > > I don't think this last requirement is possible, QEMU is as clueless > > about the capabilities of an mdev device as anyone else until that > > device is opened and probed, so how would we invent this "dummy > > device"? I don't really see how there's any ability for > > pre-determination of the device capabilities, we can only probe the > > actual device we intend to use. > > Hmm, let's say libvirt is able to create mdevs. Do the vendor drivers impose > any kind of limitations on whether a specific device-type or a specific > instance of a type does or does not present certain features like display or > migration in comparison to the other types/instances? IOW I would assume that > once the driver version does support display/migration, any mdev instance of any > mdev type the driver supports will "inherit" the support for display/migration. > If this assumption works, libvirt, knowing there are some mdev capable parent > devices, could technically create a dummy instance of the first type it can for > each parent device, passing the UUID to qemu QMP query command, qemu would then > open and probe the device, returning the capabilities which libvirt would then > cache. Next time a VM is due to start, libvirt can use the device UUID to check > the capabilities we cached and try setting appropriate config options. However, > as you've mentioned, this approach is fairly policy-driven, which doesn't cope > with what libvirt's goal is. Would such a suggestion help at all from QEMU's > POV? There is no guarantee that all mdevs are equal for a given vendor. For instance we know that the smallest vGPU instance for Intel is intended for compute offload, it's configured with barely enough framebuffer and screen resolution for a working desktop. Does it necessarily make sense that it would support all of the same capabilities as a more desktop focused mdev instance? For that matter, can we necessarily guarantee that all mdev types for a given parent device are the same class of device? For a GPU parent device we might have some VGA class devices supporting a display and some 3D controllers which don't. So I think the operative word above is "assumption". You can make whatever assumptions you want, but they're only that, there's nothing that binds the mdev vendor driver to those assumptions. > > > > The above has pressed the need for investigating some sort of > > > > alternative API through which libvirt might introspect a vfio device > > > > and with vfio device migration on the horizon, it's natural that > > > > some sort of support for migration state compatibility for the > > > > device need be considered as a second user of such an API. > > > > However, we currently have no concept of migration compatibility on > > > > a per-device level as there are no migratable devices that live > > > > outside of the QEMU code base. It's therefore assumed that per > > > > device migration compatibility is encompassed by the versioned > > > > machine type for the overall VM. We need participation all the way > > > > to the top of the VM management stack to resolve this issue and > > > > it's dragging down the (possibly) more simple question of how do we > > > > resolve the display situation. Therefore I'm looking for > > > > alternatives for display that work within what we have available to > > > > us at the moment. > > > > > > > > Erik Skultety, who initially raised the display question, has > > > > identified one possible solution, which is to simply make the > > > > display configuration the user's problem (apologies if I've > > > > misinterpreted Erik). I believe this would work something like: > > > > > > > > - libvirt identifies a version of QEMU that includes 'display' > > > > support for vfio-pci devices and defaults to adding display=off for > > > > every vfio-pci device [have we chosen the wrong default (auto) in > > > > QEMU?]. > > > > > > From libvirt's POV, having a new XML attribute display to the host > > > device type mdev should with a default value 'off', potentially > > > extending this to 'auto' once we have enough information to base our > > > decision on. We'll need to combine this with a new attribute value > > > for the