From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([208.118.235.92]:52159) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHxFH-0006kU-E3 for qemu-devel@nongnu.org; Tue, 19 Mar 2013 10:10:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1UHxF9-0007ux-9Z for qemu-devel@nongnu.org; Tue, 19 Mar 2013 10:10:31 -0400 Received: from e7.ny.us.ibm.com ([32.97.182.137]:53352) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1UHxF9-0007ut-5G for qemu-devel@nongnu.org; Tue, 19 Mar 2013 10:10:23 -0400 Received: from /spool/local by e7.ny.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Tue, 19 Mar 2013 10:10:22 -0400 Received: from d01relay06.pok.ibm.com (d01relay06.pok.ibm.com [9.56.227.116]) by d01dlp01.pok.ibm.com (Postfix) with ESMTP id 260F138C806E for ; Tue, 19 Mar 2013 10:10:20 -0400 (EDT) Received: from d01av04.pok.ibm.com (d01av04.pok.ibm.com [9.56.224.64]) by d01relay06.pok.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r2JEAJ1A13566008 for ; Tue, 19 Mar 2013 10:10:19 -0400 Received: from d01av04.pok.ibm.com (loopback [127.0.0.1]) by d01av04.pok.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r2JEAHvT018849 for ; Tue, 19 Mar 2013 10:10:18 -0400 Message-ID: <514871C2.5020108@linux.vnet.ibm.com> Date: Tue, 19 Mar 2013 10:10:10 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <1363576743-6146-1-git-send-email-mrhines@linux.vnet.ibm.com> <1363576743-6146-9-git-send-email-mrhines@linux.vnet.ibm.com> <5146D9BF.3030407@redhat.com> <51477A26.8090600@linux.vnet.ibm.com> <51482D78.3010301@redhat.com> <5148643F.2070401@linux.vnet.ibm.com> <51486733.7060207@redhat.com> <51486AD0.80309@linux.vnet.ibm.com> <51486C0D.2040609@redhat.com> In-Reply-To: <51486C0D.2040609@redhat.com> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC PATCH RDMA support v4: 08/10] introduce QEMUFileRDMA List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Paolo Bonzini Cc: aliguori@us.ibm.com, mst@redhat.com, qemu-devel@nongnu.org, owasserm@redhat.com, abali@us.ibm.com, mrhines@us.ibm.com, gokul@us.ibm.com On 03/19/2013 09:45 AM, Paolo Bonzini wrote: > This is because of downtime: You have to drain the queue anyway at the > very end, and if you don't drain it in advance after each iteration, then > the queue will have lots of bytes in it waiting for transmission and the > Virtual Machine will be stopped for a much longer period of time during > the last iteration waiting for RDMA card to finish transmission of all > those > bytes. > Shouldn't the "current chunk full" case take care of it too? > > Of course if you disable chunking you have to add a different condition, > perhaps directly into save_rdma_page. No, we don't want to flush on "chunk full" - that has a different meaning. We want to have as many chunks submitted to the hardware for transmission as possible to keep the bytes moving. >>>> 3. And also during qemu_savem_state_complete(), also using qemu_fflush. >>> This would be caught by put_buffer, but (2) would not. >>> >> I'm not sure this is good enough either - we don't want to flush >> the queue *frequently*..... only when it's necessary for performance >> .... we do want the queue to have some meat to it so the hardware >> can write bytes as fast as possible..... >> >> If we flush inside put_buffer (which is called very frequently): > Is it called at any time during RAM migration? I don't understand the question: the flushing we've been discussing is *only* for RAM migration - not for the non-live state. I haven't introduced any "new" flushes for non-live state other than when it's absolutely necessary to flush for RAM migration. >> then we have no way to distinquish *where* put buffer was called from >> (either from qemu_savevm_state_complete() or from a device-level >> function call that's using QEMUFile). > Can you make drain a no-op if there is nothing in flight? Then every > call to put_buffer after the first should not have any overhead. > > Paolo That still doesn't solve the problem: If there is nothing in flight, then there is no reason to call qemu_fflush() in the first place. This is why I avoided using fflush() in the beginning, because it sort of "confuses" who is using it: from the perspective of fflush(), you can't tell if the user calling it for RAM or for non-live state. The flushes we need are only for RAM, not the rest of it...... Make sense?