From mboxrd@z Thu Jan 1 00:00:00 1970 From: "Dr. David Alan Gilbert" Subject: Re: [Qemu-devel] live migration vs device assignment (motivation) Date: Thu, 10 Dec 2015 16:23:44 +0000 Message-ID: <20151210162343.GH2570@work-vm> References: <1448372127-28115-1-git-send-email-tianyu.lan@intel.com> <20151207165039.GA20210@redhat.com> <56685631.50700@intel.com> <20151210101840.GA2570@work-vm> <566961C1.6030000@gmail.com> <20151210114114.GE2570@work-vm> <56698E68.5040207@intel.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Cc: Yang Zhang , "Michael S. Tsirkin" , qemu-devel@nongnu.org, emil.s.tantilov@intel.com, kvm@vger.kernel.org, ard.biesheuvel@linaro.org, aik@ozlabs.ru, donald.c.skidmore@intel.com, quintela@redhat.com, eddie.dong@intel.com, nrupal.jani@intel.com, agraf@suse.de, blauwirbel@gmail.com, cornelia.huck@de.ibm.com, alex.williamson@redhat.com, kraxel@redhat.com, anthony@codemonkey.ws, amit.shah@redhat.com, pbonzini@redhat.com, mark.d.rustad@intel.com, lcapitulino@redhat.com, gerlitz.or@gmail.com To: "Lan, Tianyu" Return-path: Received: from mx1.redhat.com ([209.132.183.28]:46963 "EHLO mx1.redhat.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1750933AbbLJQXv (ORCPT ); Thu, 10 Dec 2015 11:23:51 -0500 Content-Disposition: inline In-Reply-To: <56698E68.5040207@intel.com> Sender: kvm-owner@vger.kernel.org List-ID: * Lan, Tianyu (tianyu.lan@intel.com) wrote: > > > On 12/10/2015 7:41 PM, Dr. David Alan Gilbert wrote: > >>Ideally, it is able to leave guest driver unmodified but it requires the > >>>hypervisor or qemu to aware the device which means we may need a driver in > >>>hypervisor or qemu to handle the device on behalf of guest driver. > >Can you answer the question of when do you use your code - > > at the start of migration or > > just before the end? > > Just before stopping VCPU in this version and inject VF mailbox irq to > notify the driver if the irq handler is installed. > Qemu side also will check this via the faked PCI migration capability > and driver will set the status during device open() or resume() callback. OK, hmm - I can see that would work in some cases; but: a) It wouldn't work if the guest was paused, the management can pause it before starting migration or during migration - so you might need to hook the pause as well; so that's a bit complicated. b) How long does qemu wait for the guest to respond, and what does it do if the guest doesn't respond ? How do we recover? c) How much work does the guest need to do at this point? d) It would be great if we could find a more generic way of telling the guest it's about to migrate rather than via the PCI registers of one device; imagine what happens if you have a few different devices using SR-IOV, we'd have to tell them all with separate interrupts. Perhaps we could use a virtio channel or an ACPI event or something? > >>>> >It would be great if we could avoid changing the guest; but at least your guest > >>>> >driver changes don't actually seem to be that hardware specific; could your > >>>> >changes actually be moved to generic PCI level so they could be made > >>>> >to work for lots of drivers? > >>> > >>>It is impossible to use one common solution for all devices unless the PCIE > >>>spec documents it clearly and i think one day it will be there. But before > >>>that, we need some workarounds on guest driver to make it work even it looks > >>>ugly. > > Yes, so far there is not hardware migration support and it's hard to modify > bus level code. It also will block implementation on the Windows. Well, there was agraf's trick, although that's a lot more complicated at the qemu level, but it should work with no guest modifications. Michael's point about dirty page tracking is neat, I think that simplifies it a bit if it can track dirty pages. Dave > >Dave > > -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK