From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:34928) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VPAw1-0004MD-EZ for qemu-devel@nongnu.org; Thu, 26 Sep 2013 08:44:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1VPAvt-00077w-Fs for qemu-devel@nongnu.org; Thu, 26 Sep 2013 08:44:45 -0400 Received: from e23smtp07.au.ibm.com ([202.81.31.140]:39495) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1VPAvs-00077k-To for qemu-devel@nongnu.org; Thu, 26 Sep 2013 08:44:37 -0400 Received: from /spool/local by e23smtp07.au.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Thu, 26 Sep 2013 22:44:17 +1000 Received: from d23relay03.au.ibm.com (d23relay03.au.ibm.com [9.190.235.21]) by d23dlp03.au.ibm.com (Postfix) with ESMTP id 1C51B3578052 for ; Thu, 26 Sep 2013 22:44:14 +1000 (EST) Received: from d23av01.au.ibm.com (d23av01.au.ibm.com [9.190.234.96]) by d23relay03.au.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r8QCi2B256623126 for ; Thu, 26 Sep 2013 22:44:02 +1000 Received: from d23av01.au.ibm.com (localhost [127.0.0.1]) by d23av01.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id r8QCiDkH023672 for ; Thu, 26 Sep 2013 22:44:13 +1000 Message-ID: <52442C16.7000200@linux.vnet.ibm.com> Date: Thu, 26 Sep 2013 20:44:06 +0800 From: Lei Li MIME-Version: 1.0 References: <1380119568-5530-1-git-send-email-lilei@linux.vnet.ibm.com> <5242FAE8.6030807@redhat.com> In-Reply-To: <5242FAE8.6030807@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [PATCH 0/8 RFC] migration: Introduce side channel for RAM List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: aarcange@redhat.com, quintela@redhat.com, qemu-devel@nongnu.org, mrhines@linux.vnet.ibm.com, mdroth@linux.vnet.ibm.com, anthony@codemonkey.ws, lagarcia@br.ibm.com, rcj@linux.vnet.ibm.com On 09/25/2013 11:02 PM, Paolo Bonzini wrote: > Il 25/09/2013 16:32, Lei Li ha scritto: >> This RFC patch series tries to introduce a mechanism using side >> channel pipe for RAM via SCM_RIGHTS with unix domain socket >> protocol migration. >> >> This side channel will be used for the page flipping by vmsplice, >> which will be the internal mechanism for localhost migration that >> we are trying to add. The previous patch series for localhost migration >> as link, >> >> http://lists.nongnu.org/archive/html/qemu-devel/2013-08/msg02916.html >> >> After this series, will adjust the process of current migration for >> the localhost migration and involve the vmsplice based on the previous >> patch set as link above. >> >> Please let me know if it is the proper way for it or there is anything >> need to be improved. Your suggestions and comments are very welcome, and >> thanks for Paolo for his review and useful suggestions. >> >> >> Lei Li (8): >> migration-local: add pipe protocol for QEMUFileOps >> migration-local: add qemu_fopen_pipe() >> migration-local: add send_pipefd() >> migration-local: add recv_pipefd() >> QAPI: introduce magration capability unix_page_flipping >> migration: add migrate_unix_page_flipping() >> migration-unix: side channel support on unix outgoing >> migration-unix: side channel support on unix incoming >> >> Makefile.target | 1 + >> include/migration/migration.h | 3 + >> include/migration/qemu-file.h | 4 + >> migration-local.c | 247 +++++++++++++++++++++++++++++++++++++++++ >> migration-unix.c | 48 +++++++- >> migration.c | 9 ++ >> qapi-schema.json | 8 +- >> 7 files changed, 315 insertions(+), 5 deletions(-) >> create mode 100644 migration-local.c >> > Yes, this is much closer! > > There are two problems to be fixed, but it is getting there. > > First, it breaks migration from old QEMU to new QEMU, and also migration > where the source uses "unix:" and the destination uses "fd:" migration > (this should work as long as page flipping is disabled). The problem is > that recv_pipefd() "eats" one byte, and old versions of QEMU do not send > that byte. Hi Paolo, I didn't consider this, thanks for pointing it out! > The second problem is that you are not really using a side channel; you > are still using the QEMUFile and relying on the normal migration code to > send pages on the pipe. This will not be possible when you use vmsplice. Yes, you are right, and I am trying to involve the vmsplice. > > Both problems can be addressed with a single change in your approach: > always use the Unix socket QEMUFile but, if page flipping is enabled, > only transmit page addresses on the socket; page data will be on the > pipe. You can use hooks such as before_ram_iterate, save_page and > hook_ram_load to do all your customizations: send the pipe file > descriptor, read the pipe file descriptor, and use the pipe as a side > channel. > > To fix the first problem, you can use the before_ram_iterate callback to > send the fd, and the hook_ram_load callback to receive it. The > before_ram_iterate callback can write a special 8-byte record (with the > RAM_SAVE_FLAG_HOOK set) that will trigger the hook, followed by > send_pipefd(). The load_hook callback is called after the first 8-byte > record is sent, and can just do recv_pipefd(). > > To fix the second problem, and really use the pipe as a side channel, > you can use the save_page QEMUFile callback on the send side. This > callback must return RAM_SAVE_CONTROL_NOT_SUPP if page flipping is > disabled. If it is enabled, it should write another 8-byte record with > the RAM_SAVE_FLAG_HOOK bit, this time with the address of the page on > the Unix socket; then write the page data on the pipe, and return 0. On > the receive side, the 8-byte page address will once more cause the > load_hook callback to be called. This time you already have a file > descriptor, so you do not need to call recv_pipefd(): you just extract > the page address from the 8-byte record and read the page data from the > pipe. Thanks for your comprehensive suggestions, really nice ideas! > > The basis of your code will still be the socket-based QEMUFile, but > you'll need your own QEMUFile since you're adding Unix-specific > functionality. For this it is not a problem to have two copies the > QEMUFile code for sockets, one in savevm.c and one in migration-unix.c. Have two copies of the QEMUFile code for sockets, do you mean in my own QEMUFile, say QEMUFilePipe, includes both the copy of QEMUFileSocket code (like get_fd, get_buffer, writev_buffer..) and the Unix-specific functionality code that override these three hooks like your suggestions above? I guess 'migration-unix.c' you typed is 'migration-local.c', right? > It's a very small amount of code. > > Paolo > -- Lei