From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39857) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YWPJt-0005hL-Ua for qemu-devel@nongnu.org; Fri, 13 Mar 2015 09:08:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YWPJm-0006rf-Tf for qemu-devel@nongnu.org; Fri, 13 Mar 2015 09:08:05 -0400 Received: from mx1.redhat.com ([209.132.183.28]:45562) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YWPJm-0006rS-Lh for qemu-devel@nongnu.org; Fri, 13 Mar 2015 09:07:58 -0400 Date: Fri, 13 Mar 2015 13:07:40 +0000 From: "Dr. David Alan Gilbert" Message-ID: <20150313130739.GI2486@work-vm> References: <1424883128-9841-1-git-send-email-dgilbert@redhat.com> <1424883128-9841-2-git-send-email-dgilbert@redhat.com> <20150305032119.GK18072@voom.fritz.box> <20150305092138.GA2381@work-vm> <20150310010456.GC30335@voom.fritz.box> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20150310010456.GC30335@voom.fritz.box> Subject: Re: [Qemu-devel] [PATCH v5 01/45] Start documenting how postcopy works. List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: David Gibson Cc: aarcange@redhat.com, yamahata@private.email.ne.jp, quintela@redhat.com, qemu-devel@nongnu.org, amit.shah@redhat.com, pbonzini@redhat.com, yanghy@cn.fujitsu.com * David Gibson (david@gibson.dropbear.id.au) wrote: > On Thu, Mar 05, 2015 at 09:21:39AM +0000, Dr. David Alan Gilbert wrote: > > * David Gibson (david@gibson.dropbear.id.au) wrote: > > > On Wed, Feb 25, 2015 at 04:51:24PM +0000, Dr. David Alan Gilbert > > (git) wrote: > [snip] > > > > +=== Enabling postcopy === > > > > + > > > > +To enable postcopy (prior to the start of migration): > > > > + > > > > +migrate_set_capability x-postcopy-ram on > > > > + > > > > +The migration will still start in precopy mode, however issuing: > > > > + > > > > +migrate_start_postcopy > > > > + > > > > +will now cause the transition from precopy to postcopy. > > > > +It can be issued immediately after migration is started or any > > > > +time later on. Issuing it after the end of a migration is harmless. > > > > > > It's not quite clear to me what this means. Does > > > "migrate_start_postcopy" mean it will immediately transfer execution > > > and transfer any remaining pages postcopy, or does it just mean it > > > will start postcopying once the remaining data to transfer is small > > > enough? > > > > Yes; it will flip into postcopy soon after issuing that command irrespective > > of the amount of data remaining. > > > > > What's the reason for this rather awkward two stage activation of > > > postcopy? > > > > We need to keep track of the pages that are received during the precopy phase, > > and do some madvise and other setups on the destination RAM area before precopy > > starts; and so we need to know we might want to do postcopy - so we need > > to be told early. In the earliest posted version of my patches I had a > > time-limit setting and after the time limit expired QEMU would switch into > > the second phase of postcopy itself, but Paolo suggested the migrate_start_postcopy: > > > > https://lists.nongnu.org/archive/html/qemu-devel/2014-07/msg00943.html > > > > and it works out simpler anyway. > > Ok, that makes sense. > > > > > +=== Postcopy device transfer === > > > > + > > > > +Loading of device data may cause the device emulation to access guest RAM > > > > +that may trigger faults that have to be resolved by the source, as such > > > > +the migration stream has to be able to respond with page data *during* the > > > > +device load, and hence the device data has to be read from the stream completely > > > > +before the device load begins to free the stream up. This is achieved by > > > > +'packaging' the device data into a blob that's read in one go. > > > > + > > > > +Source behaviour > > > > + > > > > +Until postcopy is entered the migration stream is identical to normal > > > > +precopy, except for the addition of a 'postcopy advise' command at > > > > +the beginning, to tell the destination that postcopy might happen. > > > > +When postcopy starts the source sends the page discard data and then > > > > +forms the 'package' containing: > > > > + > > > > + Command: 'postcopy ram listen' > > > > + The device state > > > > + A series of sections, identical to the precopy streams device state stream > > > > + containing everything except postcopiable devices (i.e. RAM) > > > > + Command: 'postcopy ram run' > > > > + > > > > +The 'package' is sent as the data part of a Command: 'CMD_PACKAGED', and the > > > > +contents are formatted in the same way as the main migration stream. > > > > > > It seems to me the "ram listen", "ram run" and CMD_PACKAGED really > > > have to be used in conjuction this way, they don't really have any use > > > on their own. So why not make it all CMD_POSTCOPY_TRANSITION and have > > > the "listen" and "run" take effect implicitly at the beginning and end > > > of the device data. > > > > CMD_PACKAGED seems like something that was generally useful; it's fairly > > complicated on it's own and so it seemed best to keep it separate. > > And can you actually think of another use case for it? > > The thing that bothers me is that the "listen" and "run" operations > will not work correctly anywhere other than at the beginning and end > of the packaged blob. It feels similar to the packaged blobs that checkpointing schemes like COLO and microcheckpointing use; although they seem to craft their own wire protocol rather than sticking with a migration protocol. What they do in terms of controlling the CPU etc is certainly different (so the RUN/listen stuff is different). Dave > -- > David Gibson | I'll have my music baroque, and my code > david AT gibson.dropbear.id.au | minimalist, thank you. NOT _the_ _other_ > | _way_ _around_! > http://www.ozlabs.org/~dgibson -- Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK