From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:53325) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VRa9G-0006nD-8g for qemu-devel@nongnu.org; Thu, 03 Oct 2013 00:04:30 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VRa98-0001z4-9A for qemu-devel@nongnu.org; Thu, 03 Oct 2013 00:04:22 -0400 Received: from e23smtp09.au.ibm.com ([202.81.31.142]:54934) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VRa97-0001xg-Kh for qemu-devel@nongnu.org; Thu, 03 Oct 2013 00:04:14 -0400 Received: from /spool/local by e23smtp09.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 3 Oct 2013 14:03:50 +1000 Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [9.190.235.21]) by d23dlp03.au.ibm.com (Postfix) with ESMTP id B5F183578052 for ; Thu, 3 Oct 2013 14:03:46 +1000 (EST) Received: from d23av01.au.ibm.com (d23av01.au.ibm.com [9.190.234.96]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r9343Vod37683356 for ; Thu, 3 Oct 2013 14:03:35 +1000 Received: from d23av01.au.ibm.com (localhost [127.0.0.1]) by d23av01.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id r9343gUf020418 for ; Thu, 3 Oct 2013 14:03:42 +1000 Message-ID: <524CEC9A.6050602@linux.vnet.ibm.com> Date: Thu, 03 Oct 2013 12:03:38 +0800 From: Lei Li MIME-Version: 1.0 References: <1380119568-5530-1-git-send-email-lilei@linux.vnet.ibm.com> <5242FAE8.6030807@redhat.com> <52442C16.7000200@linux.vnet.ibm.com> In-Reply-To: <52442C16.7000200@linux.vnet.ibm.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: aarcange@redhat.com, quintela@redhat.com, mdroth@linux.vnet.ibm.com, mrhines@linux.vnet.ibm.com, qemu-devel@nongnu.org, anthony@codemonkey.ws, lagarcia@br.ibm.com, rcj@linux.vnet.ibm.com On 09/26/2013 08:44 PM, Lei Li wrote: > On 09/25/2013 11:02 PM, Paolo Bonzini wrote: >> Il 25/09/2013 16:32, Lei Li ha scritto: >>> This RFC patch series tries to introduce a mechanism using side >>> channel pipe for RAM via SCM_RIGHTS with unix domain socket >>> protocol migration. >>> >>> This side channel will be used for the page flipping by vmsplice, >>> which will be the internal mechanism for localhost migration that >>> we are trying to add. The previous patch series for localhost migration >>> as link, >>> >>> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html >>> >>> After this series, will adjust the process of current migration for >>> the localhost migration and involve the vmsplice based on the previous >>> patch set as link above. >>> >>> Please let me know if it is the proper way for it or there is anything >>> need to be improved. Your suggestions and comments are very welcome, >>> and >>> thanks for Paolo for his review and useful suggestions. >>> >>> >>> Lei Li (8): >>> migration-local: add pipe protocol for QEMUFileOps >>> migration-local: add qemu_fopen_pipe() >>> migration-local: add send_pipefd() >>> migration-local: add recv_pipefd() >>> QAPI: introduce magration capability unix_page_flipping >>> migration: add migrate_unix_page_flipping() >>> migration-unix: side channel support on unix outgoing >>> migration-unix: side channel support on unix incoming >>> >>> Makefile.target | 1 + >>> include/migration/migration.h | 3 + >>> include/migration/qemu-file.h | 4 + >>> migration-local.c | 247 >>> +++++++++++++++++++++++++++++++++++++++++ >>> migration-unix.c | 48 +++++++- >>> migration.c | 9 ++ >>> qapi-schema.json | 8 +- >>> 7 files changed, 315 insertions(+), 5 deletions(-) >>> create mode 100644 migration-local.c >>> >> Yes, this is much closer! >> >> There are two problems to be fixed, but it is getting there. >> >> First, it breaks migration from old QEMU to new QEMU, and also migration >> where the source uses "unix:" and the destination uses "fd:" migration >> (this should work as long as page flipping is disabled). The problem is >> that recv_pipefd() "eats" one byte, and old versions of QEMU do not send >> that byte. > > Hi Paolo, > > I didn't consider this, thanks for pointing it out! > >> The second problem is that you are not really using a side channel; you >> are still using the QEMUFile and relying on the normal migration code to >> send pages on the pipe. This will not be possible when you use >> vmsplice. > > Yes, you are right, and I am trying to involve the vmsplice. > >> >> Both problems can be addressed with a single change in your approach: >> always use the Unix socket QEMUFile but, if page flipping is enabled, >> only transmit page addresses on the socket; page data will be on the >> pipe. You can use hooks such as before_ram_iterate, save_page and >> hook_ram_load to do all your customizations: send the pipe file >> descriptor, read the pipe file descriptor, and use the pipe as a side >> channel. >> >> To fix the first problem, you can use the before_ram_iterate callback to >> send the fd, and the hook_ram_load callback to receive it. The >> before_ram_iterate callback can write a special 8-byte record (with the >> RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by >> send_pipefd(). The load_hook callback is called after the first 8-byte >> record is sent, and can just do recv_pipefd(). Hi Paolo, When debugging the code, I realized that this problem might still exist. In the incoming part, it will qemu_fopen_pipe() in unix_accept_incoming_migration first to enable the load_hook callback, the check action of this RAM_SAVE_FLAG_HOOK flags would lead to 8 bytes taken. Turns out, it will break normal unix migration (without unix-page-flipping), because no matter normal unix migration or unix-page-flipping migration, the incoming side has to check this 8-byes flags first to decide whether the load_hook is called, and normal unix migration did not send this 8-byte flags. I wonder if I didn't understand your suggestion correctly? >> >> To fix the second problem, and really use the pipe as a side channel, >> you can use the save_page QEMUFile callback on the send side. This >> callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is >> disabled. If it is enabled, it should write another 8-byte record with >> the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on >> the Unix socket; then write the page data on the pipe, and return 0. On >> the receive side, the 8-byte page address will once more cause the >> load_hook callback to be called. This time you already have a file >> descriptor, so you do not need to call recv_pipefd(): you just extract >> the page address from the 8-byte record and read the page data from the >> pipe. > > Thanks for your comprehensive suggestions, really nice ideas! > >> >> The basis of your code will still be the socket-based QEMUFile, but >> you'll need your own QEMUFile since you're adding Unix-specific >> functionality. For this it is not a problem to have two copies the >> QEMUFile code for sockets, one in savevm.c and one in migration-unix.c. > > Have two copies of the QEMUFile code for sockets, do you mean in my own > QEMUFile, say QEMUFilePipe, includes both the copy of QEMUFileSocket > code (like get_fd, get_buffer, writev_buffer..) and the Unix-specific > functionality code that override these three hooks like your suggestions > above? > > I guess 'migration-unix.c' you typed is 'migration-local.c', right? > >> It's a very small amount of code. >> >> Paolo >> > > -- Lei