From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1MTd5G-0002dg-43 for qemu-devel@nongnu.org; Wed, 22 Jul 2009 10:46:18 -0400 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1MTd5B-0002d7-Bt for qemu-devel@nongnu.org; Wed, 22 Jul 2009 10:46:17 -0400 Received: from [199.232.76.173] (port=52272 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1MTd5B-0002d4-5v for qemu-devel@nongnu.org; Wed, 22 Jul 2009 10:46:13 -0400 Received: from mail2.shareable.org ([80.68.89.115]:35458) by monty-python.gnu.org with esmtps (TLS-1.0:RSA_AES_256_CBC_SHA1:32) (Exim 4.60) (envelope-from ) id 1MTd5A-0004bc-JW for qemu-devel@nongnu.org; Wed, 22 Jul 2009 10:46:12 -0400 Date: Wed, 22 Jul 2009 15:46:07 +0100 From: Jamie Lokier Subject: Re: [Qemu-devel] CPUID feature bits not saved with migration Message-ID: <20090722144607.GA24303@shareable.org> References: <4A5DD0B0.7070700@amd.com> <4A5DE9E5.2080809@codemonkey.ws> <20090715151234.GA28724@shareable.org> <4A5EFC16.7070808@amd.com> <20090716142809.GA16461@shareable.org> <4A671323.2070604@amd.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4A671323.2070604@amd.com> List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Andre Przywara Cc: qemu-devel@nongnu.org Andre Przywara wrote: > (Sorry for the late reply, I had some mail troubles) > > Jamie Lokier wrote: > >Andre Przywara wrote: > >>Jamie Lokier wrote: > >>>Anthony Liguori wrote: > >>>>It's unclear what to do about -cpu host. If we did migrate cpuid > >>>>values, then -cpu would effectively be ignored after an incoming > >>>>migration. > >>>The new host might not support all the cpuid features of the old host, > >>>whether by -cpu host or explicit cpuid. What happens then? > >>If you plan to migrate, you should think of this in advance. > > > >In my experience so far, for small sites, you don't plan migration for > >2-3 years after starting the VM, because it's only then you realise > >your host hardware is quite old, you buy a replacement to consolidate, > >and you are still running the VM that you didn't know would still be > >mission critical years later. > > That is one use-case for live migration. Another would be a migration > pool with lots of machines each running some VMs. If one host is loaded, > you can migrate to a lesser loaded one. Think of a hoster or cloud like > environment. I realise that, and I was objecting to the apparent assumption that a pool type environment is the only use-case for migration. > >At least, that's been my experience so far. I've "cold migrated" a > >few VMs, and in some cases from non-virtual machines to VMs. None of > >these could be planned when the guest was first installed, especially > >the ones where it wasn't realised the guest would outlive the host > >hardware. > Fortunately it seems like that newer CPUs only _add_ CPUID bits, so this > should not be a problem. Again, not my experience. They only add CPUID bits when you buy CPUs from the same manufacturer. Ok, there's only three manufacturers, but they are different. > >I have to say, unfortunately hot migration has never been an option > >because the version of KVM running on the new host is invariably > >incompatible with the KVM running on the old host. > So far I have only seen problems like this if the target host KVM > version is older than the source one. Some of these issues could be > overcome by putting a translator application between source and target, > but I am not sure whether the effort is worth the results. > What kind of issues do you see? Are you migrating from newer KVMs to > older ones? I've never migrated to an older KVM. Or to be honest, to a newer one. I've tried loadvm of a previous savevm, and that didn't work from an older KVM to a newer one. Since then I think I've understood from on this list that cross-version migration (loadvm or migrate) in either direction is not supported, is not worth supporting, and should not be expected to work. > >But if guest configuration is ever included in the saved state for > >migration, migration will really easy. I hope it's just as easy to do > >"cold migration". > Agreed. We should have a savevm section transferring the guest config. I'm glad it's not just me :-) > >Async: Do we save RAM state across reboots normally? I know of OSes > >which expect at least some part of RAM to survive reboots, so killing > >the VM and restarting on another host would change the behaviour, > >compared with rebooting locally; that's not transparent migration, > >it's a subtle, unexpected behaviour change. Unfortunately doing the > >right thing involves savev, which pauses the guests for a long time. > >The pause comes from saving and loading RAM, something which migration > >handles well. > Have you seen any real life problems with this? What are these OSes? I've wanted to periodically snapshot a live server while it was running, as a sort of backup against screwups, and I tried using savevm. It was running Windows Server 2003 at the time; now it's Server 2008, and the host is Ubuntu Linux 8.10 with KVM built from source. But I don't think any of those versions are relevant. It was unacceptable to invoke "savevm" periodically, because it took some 20 seconds or more (I don't remember exactly) with the VM paused; that was a live server. I'm guessing that must have been the time to save the RAM contents. >>From that I concluded that savevm+loadvm would take >40 seconds of downtime to transfer across hosts. Which isn't awful, but using the migration facility would clearly be much less downtime. > >There's also the small matter of migration having a totally different > >interface compared with savevm right now, with savevm requiring a > >dummy qcow2 disk while migration transfers control across a network > >with no temporary file. > You are right, that is pretty unfortunate. I worked around this > limitation by using the exec: prefix with migrate to let a shell script > dump the migration stream to disk, with the same trick you can reload > the state again. That worked pretty well for me in the past. That's nice; I like that. Maybe savevm could just use it ;-) > BTW, do you know of any x86 machines which really allow physical CPU > hotplugging? I have the impression the ES7000 does, but that's impressions from mailing list postings which never made it completely clear. -- Jamie