From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mailman by lists.gnu.org with tmda-scanned (Exim 4.43) id 1NCDyQ-0006HC-PH for qemu-devel@nongnu.org; Sun, 22 Nov 2009 10:03:34 -0500 Received: from exim by lists.gnu.org with spam-scanned (Exim 4.43) id 1NCDyM-0006Fm-8E for qemu-devel@nongnu.org; Sun, 22 Nov 2009 10:03:34 -0500 Received: from [199.232.76.173] (port=37425 helo=monty-python.gnu.org) by lists.gnu.org with esmtp (Exim 4.43) id 1NCDyM-0006Fg-4v for qemu-devel@nongnu.org; Sun, 22 Nov 2009 10:03:30 -0500 Received: from mx1.redhat.com ([209.132.183.28]:59421) by monty-python.gnu.org with esmtp (Exim 4.60) (envelope-from ) id 1NCDyL-0002Lk-G9 for qemu-devel@nongnu.org; Sun, 22 Nov 2009 10:03:29 -0500 Received: from int-mx03.intmail.prod.int.phx2.redhat.com (int-mx03.intmail.prod.int.phx2.redhat.com [10.5.11.16]) by mx1.redhat.com (8.13.8/8.13.8) with ESMTP id nAMF3SjL006380 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK) for ; Sun, 22 Nov 2009 10:03:28 -0500 Received: from [10.35.0.61] (dhcp-0-61.tlv.redhat.com [10.35.0.61]) by int-mx03.intmail.prod.int.phx2.redhat.com (8.13.8/8.13.8) with ESMTP id nAMF3QpP031445 for ; Sun, 22 Nov 2009 10:03:27 -0500 Message-ID: <4B0952C9.9010803@redhat.com> Date: Sun, 22 Nov 2009 17:03:37 +0200 From: Dor Laor MIME-Version: 1.0 Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: [Qemu-devel] Live migration protocol, device features, ABIs and other beasts Reply-To: dlaor@redhat.com List-Id: qemu-devel.nongnu.org List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: qemu-devel In the last couple of days we discovered some issues regarding stable ABI and the robustness of the live migration protocol. Let's just jump right into it, ordered by complexity: 1. Control *every* feature exposed to the guest by qemu cmdline: While thinking on cross version migration, and reviewing some patches, I noticed that there are many times that we use feature bits in order to expose functionality for the guest driver - example: VIRTIO_BLK_F_BARRIER, but we do not control it from qemu cmdline. The result is that guest running on a newer qemu cannot live migrate into older qemu without the barrier feature. Like this barrier example, there are probably many cases that we do keep device/driver abi but forget new/old release abi. The solution here is simpler - Every guest visible change should translate into cmdline option. This is part of the machine type and in addition should be configurable. It's an issue we all should keep in the back of our heads and popup when a new capability/change are introduced. 2. Live migration inherent problem. Currently, even with VMState, the protocol is not flexible enough. We run into problem when we needed to fix pvclock migration issue. The fix included 2 additional fields in save/load state and thus needed a new version number. The trouble is that the load function does not accept sections with versions greater than the one it supports. We cannot even create a new 'hack section' for new code since the sections are ordered and expected to be exact match on the destination. The result is that new->old migration cannot work. This is not cross releases even! It means that even a small bug in current release prevents live migration between various instances of the code. It forces us to decide whether to fix pvclock migration issue vs allow new->old migration. Another ugly hack is to add cmdline that will control this behavior. Still it's a pain to mgmt stack and users. The solution here is more complex. One can claim that we should allow newer sections to be accepted by current code (and send the section size) and send optional sections. This would be a nasty work around. IMHO we should 'specify' the migration protocol and introduce capabilities, feature bits, etc. This way we'll have a robust, extensible protocol that will withstand any potential issue. Both Michael Tisrkin and I suggest it at the time vmstate was introduced. Vmstate is good for the code but it's not a protocol. Which protocol should we use? You're smarter than me, please suggest one. wrt the above guest abi issue, we should write a qemu spec with clear definitions for devices, drivers, versions, etc. Looking forward interesting fruitful discussion, Dor