From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37617) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uo4NO-0001Rd-I4 for qemu-devel@nongnu.org; Sun, 16 Jun 2013 00:15:41 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Uo4NH-000680-W6 for qemu-devel@nongnu.org; Sun, 16 Jun 2013 00:15:38 -0400 Received: from e38.co.us.ibm.com ([32.97.110.159]:57302) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Uo4NH-00067R-ME for qemu-devel@nongnu.org; Sun, 16 Jun 2013 00:15:31 -0400 Received: from /spool/local by e38.co.us.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Sat, 15 Jun 2013 22:15:29 -0600 Received: from d03relay02.boulder.ibm.com (d03relay02.boulder.ibm.com [9.17.195.227]) by d03dlp02.boulder.ibm.com (Postfix) with ESMTP id A6B033E40040 for ; Sat, 15 Jun 2013 22:13:29 -0600 (MDT) Received: from d03av04.boulder.ibm.com (d03av04.boulder.ibm.com [9.17.195.170]) by d03relay02.boulder.ibm.com (8.13.8/8.13.8/NCO v10.0) with ESMTP id r5G4Dmc7151160 for ; Sat, 15 Jun 2013 22:13:48 -0600 Received: from d03av04.boulder.ibm.com (loopback [127.0.0.1]) by d03av04.boulder.ibm.com (8.14.4/8.13.1/NCO v10.0 AVout) with ESMTP id r5G4DlHE028137 for ; Sat, 15 Jun 2013 22:13:47 -0600 Message-ID: <51BD3B7A.4050504@linux.vnet.ibm.com> Date: Sun, 16 Jun 2013 00:13:46 -0400 From: "Michael R. Hines" MIME-Version: 1.0 References: <1370880226-2208-1-git-send-email-mrhines@linux.vnet.ibm.com> <4168C988EBDF2141B4E0B6475B6A73D10CE2AAC1@G6W2488.americas.hpqcorp.net> <51B60ABA.2070401@linux.vnet.ibm.com> <4168C988EBDF2141B4E0B6475B6A73D10CE2BAE7@G6W2488.americas.hpqcorp.net> <51B7B652.3070905@linux.vnet.ibm.com> <51B85EE5.1050702@hp.com> <51B868B3.9090607@linux.vnet.ibm.com> <51B9A614.2050101@hp.com> <51B9C2D6.30000@linux.vnet.ibm.com> <51B9D6A8.9070007@hp.com> <51B9DD5C.1030409@linux.vnet.ibm.com> <51BA264E.4010701@redhat.com> <51BA36DD.8050703@linux.vnet.ibm.com> <51BAB892.8000601@linux.vnet.ibm.com> <51BB7F62.9000304@linux.vnet.ibm.com> In-Reply-To: <51BB7F62.9000304@linux.vnet.ibm.com> Content-Type: multipart/mixed; boundary="------------050404090102000408040405" Subject: Re: [Qemu-devel] RDMA: please pull and re-test freezing fixes List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: "Michael R. Hines" Cc: "mrhines@us.ibm.com" , Bulent Abali , "qemu-devel@nongnu.org" , Anthony Liguori , Juan Jose Quintela Carreira This is a multi-part message in MIME format. --------------050404090102000408040405 Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit These are great results. Even the one that doesn't converge is good, because it shows that your patch for CPU throttling earlier on the mailing list is still very necessary. Thanks a lot, I will send out a V10 patch to everybody's results. - Michael Would you be interested in me posting them to the wiki? On 6/14/2013 1:38 PM, Michael R. Hines wrote: > Chegu, > > I sent a V9 to the mailing list: > > The version goes even further, by explicitly timing the pinning > latency and > pushing the value out to QMP so the user clearly knows which component > of total migration time is consumed by pinning. > > If you're satisfied, I'd appreciate if I could add your Reviewed-By: =) > Pl. see below...and yes you can add me. Thanks, Vinod The migration speed was set to 40G and the downtime to 2sec for all experiments below. Note: Idle guests are not interesting due to tons of zero pages etc...but including them here to highlght the overhead of pinning. 1) 20vcpu/64GB guest: (kind of a larger sized Cloud-type guest) a) Idle guest with No pinning (default) : capabilities: xbzrle: off x-rdma-pin-all: off Migration status: completed total time: 51062 milliseconds downtime: 1948 milliseconds pin-all: 0 milliseconds transferred ram: 1816547 kbytes throughput: 6872.23 mbps remaining ram: 0 kbytes total ram: 67117632 kbytes duplicate: 16331552 pages skipped: 0 pages normal: 450038 pages normal bytes: 1800152 kbytes b) Idle guest with Pinning : capabilities: xbzrle: off x-rdma-pin-all: on Migration status: completed total time: 47451 milliseconds downtime: 2639 milliseconds pin-all: 22780 milliseconds transferred ram: 67136643 kbytes throughput: 25222.91 mbps remaining ram: 0 kbytes total ram: 67117632 kbytes duplicate: 0 pages skipped: 0 pages normal: 16780064 pages normal bytes: 67120256 kbytes There weere no freezes observed in the guest at the start of the migration but the qemu monitor prompt was not responsive for the duration of the memory pinning. Total migration time was affected by the cost pinning at the start of the migration as shown above( This issue can be pursued and optimized later). c) Pining + guest running a Java warehouse workload (I cranked the workload up to keep the guest 95+% busy) capabilities: xbzrle: off x-rdma-pin-all: on Migration status: active total time: 412706 milliseconds expected downtime: 499 milliseconds pin-all: 22758 milliseconds transferred ram: 657243669 kbytes throughput: 25241.89 mbps remaining ram: 7281848 kbytes total ram: 67117632 kbytes duplicate: 0 pages skipped: 0 pages normal: 164270810 pages normal bytes: 657083240 kbytes dirty pages rate: 369925 pages No Convergence ! (For workloads where the memory dirty rate is very high there are other alternatives that have been discussed in the past...) --- Enterprise type guests tend to get fatter (more memory per cpu) than the larger Cloud guests...so here are a coupld of them. a) 20VCPU/256G Idle guest : Default: capabilities: xbzrle: off x-rdma-pin-all: off Migration status: completed total time: 259259 milliseconds downtime: 3924 milliseconds pin-all: 0 milliseconds transferred ram: 5522078 kbytes throughput: 6586.06 mbps remaining ram: 0 kbytes total ram: 268444224 kbytes duplicate: 65755168 pages skipped: 0 pages normal: 1364124 pages normal bytes: 5456496 kbytes Pinned: capabilities: xbzrle: off x-rdma-pin-all: on Migration status: completed total time: 219053 milliseconds downtime: 4277 milliseconds pin-all: 118153 milliseconds transferred ram: 268512809 kbytes throughput: 22209.32 mbps remaining ram: 0 kbytes total ram: 268444224 kbytes duplicate: 0 pages skipped: 0 pages normal: 67111817 pages normal bytes: 268447268 kbytes b) 40VCPU/512GB Idle guest : Default: capabilities: xbzrle: off x-rdma-pin-all: off Migration status: completed total time: 670577 milliseconds downtime: 6139 milliseconds pin-all: 0 milliseconds transferred ram: 10279256 kbytes throughput: 6150.93 mbps remaining ram: 0 kbytes total ram: 536879680 kbytes duplicate: 131704099 pages skipped: 0 pages normal: 2537017 pages normal bytes: 10148068 kbytes Pinned: capabilities: xbzrle: off x-rdma-pin-all: on Migration status: completed total time: 527576 milliseconds downtime: 6314 milliseconds pin-all: 312984 milliseconds transferred ram: 537129685 kbytes throughput: 20177.27 mbps remaining ram: 0 kbytes total ram: 536879680 kbytes duplicate: 0 pages skipped: 0 pages normal: 134249644 pages normal bytes: 536998576 kbytes No freezes in the guest due to memory pinning. (Freezes were only due to the dirty bitmap synchup stuff which is being done while BQL is held. Juan is working on addresing already for qemu 1.6) --------------050404090102000408040405 Content-Type: text/plain; charset=UTF-8; name="01" Content-Transfer-Encoding: 7bit Content-Disposition: attachment; filename="01" The migration speed was set to 40G and the downtime to 2sec for all experiments below. Note: Idle guests are not interesting due to tons of zero pages etc...but including them here to highlght the overhead of pinning. 1) 20vcpu/64GB guest: (kind of a larger sized Cloud-type guest) a) Idle guest with No pinning (default) : capabilities: xbzrle: off x-rdma-pin-all: off Migration status: completed total time: 51062 milliseconds downtime: 1948 milliseconds pin-all: 0 milliseconds transferred ram: 1816547 kbytes throughput: 6872.23 mbps remaining ram: 0 kbytes total ram: 67117632 kbytes duplicate: 16331552 pages skipped: 0 pages normal: 450038 pages normal bytes: 1800152 kbytes b) Idle guest with Pinning : capabilities: xbzrle: off x-rdma-pin-all: on Migration status: completed total time: 47451 milliseconds downtime: 2639 milliseconds pin-all: 22780 milliseconds transferred ram: 67136643 kbytes throughput: 25222.91 mbps remaining ram: 0 kbytes total ram: 67117632 kbytes duplicate: 0 pages skipped: 0 pages normal: 16780064 pages normal bytes: 67120256 kbytes There weere no freezes observed in the guest at the start of the migration but the qemu monitor prompt was not responsive for the duration of the memory pinning. Total migration time was affected by the cost pinning at the start of the migration as shown above( This issue can be pursued and optimized later). c) Pining + guest running a Java warehouse workload (I cranked the workload up to keep the guest 95+% busy) capabilities: xbzrle: off x-rdma-pin-all: on Migration status: active total time: 412706 milliseconds expected downtime: 499 milliseconds pin-all: 22758 milliseconds transferred ram: 657243669 kbytes throughput: 25241.89 mbps remaining ram: 7281848 kbytes total ram: 67117632 kbytes duplicate: 0 pages skipped: 0 pages normal: 164270810 pages normal bytes: 657083240 kbytes dirty pages rate: 369925 pages No Convergence ! (For workloads where the memory dirty rate is very high there are other alternatives that have been discussed in the past...) --- Enterprise type guests tend to get fatter (more memory per cpu) than the larger Cloud guests...so here are a coupld of them. a) 20VCPU/256G Idle guest : Default: capabilities: xbzrle: off x-rdma-pin-all: off Migration status: completed total time: 259259 milliseconds downtime: 3924 milliseconds pin-all: 0 milliseconds transferred ram: 5522078 kbytes throughput: 6586.06 mbps remaining ram: 0 kbytes total ram: 268444224 kbytes duplicate: 65755168 pages skipped: 0 pages normal: 1364124 pages normal bytes: 5456496 kbytes Pinned: capabilities: xbzrle: off x-rdma-pin-all: on Migration status: completed total time: 219053 milliseconds downtime: 4277 milliseconds pin-all: 118153 milliseconds transferred ram: 268512809 kbytes throughput: 22209.32 mbps remaining ram: 0 kbytes total ram: 268444224 kbytes duplicate: 0 pages skipped: 0 pages normal: 67111817 pages normal bytes: 268447268 kbytes b) 40VCPU/512GB Idle guest : Default: capabilities: xbzrle: off x-rdma-pin-all: off Migration status: completed total time: 670577 milliseconds downtime: 6139 milliseconds pin-all: 0 milliseconds transferred ram: 10279256 kbytes throughput: 6150.93 mbps remaining ram: 0 kbytes total ram: 536879680 kbytes duplicate: 131704099 pages skipped: 0 pages normal: 2537017 pages normal bytes: 10148068 kbytes Pinned: capabilities: xbzrle: off x-rdma-pin-all: on Migration status: completed total time: 527576 milliseconds downtime: 6314 milliseconds pin-all: 312984 milliseconds transferred ram: 537129685 kbytes throughput: 20177.27 mbps remaining ram: 0 kbytes total ram: 536879680 kbytes duplicate: 0 pages skipped: 0 pages normal: 134249644 pages normal bytes: 536998576 kbytes No freezes in the guest due to memory pinning. (Freezes were only due to the dirty bitmap synchup stuff which is being done while BQL is held. Juan is working on addresing already for qemu 1.6) --------------050404090102000408040405--