From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:55239) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHxQi-00015h-9C for qemu-devel@nongnu.org; Tue, 19 Mar 2013 10:22:24 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UHxQg-0003Ru-Ry for qemu-devel@nongnu.org; Tue, 19 Mar 2013 10:22:20 -0400 Received: from mx1.redhat.com ([209.132.183.28]:44426) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHxQg-0003Rn-Jp for qemu-devel@nongnu.org; Tue, 19 Mar 2013 10:22:18 -0400 Message-ID: <5148748C.5050509@redhat.com> Date: Tue, 19 Mar 2013 15:22:04 +0100 From: Paolo Bonzini MIME-Version: 1.0 References: <1363576743-6146-1-git-send-email-mrhines@linux.vnet.ibm.com> <1363576743-6146-9-git-send-email-mrhines@linux.vnet.ibm.com> <5146D9BF.3030407@redhat.com> <51477A26.8090600@linux.vnet.ibm.com> <51482D78.3010301@redhat.com> <5148643F.2070401@linux.vnet.ibm.com> <51486733.7060207@redhat.com> <51486AD0.80309@linux.vnet.ibm.com> <51486C0D.2040609@redhat.com> <514871C2.5020108@linux.vnet.ibm.com> In-Reply-To: <514871C2.5020108@linux.vnet.ibm.com> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v4: 08/10] introduce QEMUFileRDMA List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael R. Hines" Cc: aliguori@us.ibm.com, mst@redhat.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com Il 19/03/2013 15:10, Michael R. Hines ha scritto: > On 03/19/2013 09:45 AM, Paolo Bonzini wrote: >> This is because of downtime: You have to drain the queue anyway at the >> very end, and if you don't drain it in advance after each iteration, then >> the queue will have lots of bytes in it waiting for transmission and the >> Virtual Machine will be stopped for a much longer period of time during >> the last iteration waiting for RDMA card to finish transmission of all >> those >> bytes. >> Shouldn't the "current chunk full" case take care of it too? >> >> Of course if you disable chunking you have to add a different condition, >> perhaps directly into save_rdma_page. > > No, we don't want to flush on "chunk full" - that has a different meaning. > We want to have as many chunks submitted to the hardware for transmission > as possible to keep the bytes moving. That however gives me an idea... Instead of the full drain at the end of an iteration, does it make sense to do a "partial" drain at every chunk full, so that you don't have > N bytes pending and the downtime is correspondingly limited? >>>>> 3. And also during qemu_savem_state_complete(), also using >>>>> qemu_fflush. >>>> This would be caught by put_buffer, but (2) would not. >>>> >>> I'm not sure this is good enough either - we don't want to flush >>> the queue *frequently*..... only when it's necessary for performance >>> .... we do want the queue to have some meat to it so the hardware >>> can write bytes as fast as possible..... >>> >>> If we flush inside put_buffer (which is called very frequently): >> Is it called at any time during RAM migration? > > I don't understand the question: the flushing we've been discussing > is *only* for RAM migration - not for the non-live state. Yes. But I would like to piggyback the final, full drain on the switch from RAM migration to device migration. >> Can you make drain a no-op if there is nothing in flight? Then every >> call to put_buffer after the first should not have any overhead. > > That still doesn't solve the problem: If there is nothing in flight, > then there is no reason to call qemu_fflush() in the first place. If there is no RAM migration in flight. So you have migrate RAM ... RAM migration finished, device migration start put_buffer <<<<< QEMUFileRDMA triggers drain put_buffer put_buffer put_buffer ... > The flushes we need are only for RAM, not the rest of it...... > > Make sense? Paolo