From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id E1D5BC433FE for ; Mon, 7 Nov 2022 15:31:08 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 672D64B88F; Mon, 7 Nov 2022 10:31:08 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@kernel.org Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id CGyn6hycfmVh; Mon, 7 Nov 2022 10:31:07 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 282A74B893; Mon, 7 Nov 2022 10:31:07 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id A03BE4B88C for ; Mon, 7 Nov 2022 10:31:05 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id TWGDRT5lm5Ke for ; Mon, 7 Nov 2022 10:31:04 -0500 (EST) Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 6890D4B885 for ; Mon, 7 Nov 2022 10:31:04 -0500 (EST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 4144B60DE3; Mon, 7 Nov 2022 15:31:03 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A11DCC433D6; Mon, 7 Nov 2022 15:31:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1667835062; bh=bkVktwW5TQjNJ7vT92h9Ndj7XiwAcQLTowe267DgFCM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=n6poy86h9Rm4cDZcXm40fl1O2bbwobQTLE1dNyZvG7LyEe4N6BXXkjxlhywPpw+sW cXAM33t1k16JO/VLniR7wxF3VPJNxL2/XhIcq5c/vX/xxkDnyUBcco1opwqdd8sUA7 Ll5SjjbNQHHFgXCSX3Y/Kau8KyhSEhuKlCTlUTZtA9buC06CBQyomUoxp1ByWAtRry wTPhEApW2RUD+v3HYx2zGnXe24Yw+65Y1VvGD0jOv67zq94Kdjt8I13O3ITOsEGnd0 lY6An1wsYTUm03vJGthMwUcLa3iE0PmMzH6BZ/AMXwY8zWWwSpe3RubmzrgrbtFA6U m7nVDvTlb/y7w== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1os45U-004QlV-B6; Mon, 07 Nov 2022 15:31:00 +0000 Date: Mon, 07 Nov 2022 15:30:59 +0000 Message-ID: <86zgd2pyrw.wl-maz@kernel.org> From: Marc Zyngier To: Peter Xu Subject: Re: [PATCH v8 3/7] KVM: Support dirty ring in conjunction with bitmap In-Reply-To: References: <20221104234049.25103-1-gshan@redhat.com> <20221104234049.25103-4-gshan@redhat.com> <87o7tkf5re.wl-maz@kernel.org> <87iljrg7vd.wl-maz@kernel.org> <867d07qfvk.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: peterx@redhat.com, gshan@redhat.com, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, shuah@kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, ajones@ventanamicro.com, bgardon@google.com, dmatlack@google.com, will@kernel.org, suzuki.poulose@arm.com, alexandru.elisei@arm.com, pbonzini@redhat.com, seanjc@google.com, oliver.upton@linux.dev, zhenyzha@redhat.com, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Cc: kvm@vger.kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, dmatlack@google.com, will@kernel.org, shan.gavin@gmail.com, bgardon@google.com, kvmarm@lists.linux.dev, pbonzini@redhat.com, zhenyzha@redhat.com, shuah@kernel.org, kvmarm@lists.cs.columbia.edu, ajones@ventanamicro.com X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Mon, 07 Nov 2022 14:59:41 +0000, Peter Xu wrote: > > On Mon, Nov 07, 2022 at 09:21:35AM +0000, Marc Zyngier wrote: > > On Sun, 06 Nov 2022 21:06:43 +0000, > > Peter Xu wrote: > > > > > > It's definitely not the case for x86, but if it's true for > > > arm64, then could the DMA be spread across all the guest pages? > > > If it's also true, I really don't know how this will work.. > > > > Of course, all pages can be the target of DMA. It works the same way > > it works for the ITS: you sync the state, you obtain the dirty bits, > > you move on. > > > > And mimicking what x86 does is really not my concern (if you still > > think that arm64 is just another flavour of x86, stay tuned! ;-). > > I didn't mean so, I should probably stop mentioning x86. :) Please! I turned off my last x86 development machine over the weekend, and my x86 laptop is now a glorified window manager... ;-) > I had some sense already from the topics in past few years of kvm forum. > Yeah I'll be looking forward to anything more coming. Yup. Hopefully we won't have to wait for too long to see this stuff (I had good discussions on the subject at both KF and Plumbers in Dublin earlier this year). > > > We're only syncing the dirty bitmap once right now with the > > > protocol. If that can cover most of the guest mem, it's same as > > > non-live. If we sync it periodically, then it's the same as > > > enabling dirty-log alone and the rings are useless. > > > > I'm glad that you finally accept it: the ring *ARE* useless in the > > general sense. Only limited, CPU-only workloads can make any use of > > the current design. This probably covers a large proportion of what > > the cloud vendors do, but this doesn't work for general situations > > where you have a stream of dirty pages originating outside of the > > CPUs. > > The ring itself is really not the thing to blame, IMHO it's a good attempt > to try out de-coupling guest size in regard of dirty tracking from kvm. It > may not be perfect, but it may still service some of the goals, e.g., at > least it allows the user app to detect per-vcpu information and also since > there's the ring full events we can do something more than before like the > vcpu throttling that China Telecom does with the ring structures. I don't disagree with that: for vcpu-based workloads, the rings are great and doing their job. It's just that there is another side to this problem, and you'll have to deal with both eventually. We're just ahead of the curve here... Thanks, M. -- Without deviation from the norm, progress is not possible. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from smtp.kernel.org (aws-us-west-2-korg-mail-1.web.codeaurora.org [10.30.226.201]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id D4EAFD527 for ; Mon, 7 Nov 2022 15:31:02 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A11DCC433D6; Mon, 7 Nov 2022 15:31:02 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1667835062; bh=bkVktwW5TQjNJ7vT92h9Ndj7XiwAcQLTowe267DgFCM=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=n6poy86h9Rm4cDZcXm40fl1O2bbwobQTLE1dNyZvG7LyEe4N6BXXkjxlhywPpw+sW cXAM33t1k16JO/VLniR7wxF3VPJNxL2/XhIcq5c/vX/xxkDnyUBcco1opwqdd8sUA7 Ll5SjjbNQHHFgXCSX3Y/Kau8KyhSEhuKlCTlUTZtA9buC06CBQyomUoxp1ByWAtRry wTPhEApW2RUD+v3HYx2zGnXe24Yw+65Y1VvGD0jOv67zq94Kdjt8I13O3ITOsEGnd0 lY6An1wsYTUm03vJGthMwUcLa3iE0PmMzH6BZ/AMXwY8zWWwSpe3RubmzrgrbtFA6U m7nVDvTlb/y7w== Received: from sofa.misterjones.org ([185.219.108.64] helo=goblin-girl.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1os45U-004QlV-B6; Mon, 07 Nov 2022 15:31:00 +0000 Date: Mon, 07 Nov 2022 15:30:59 +0000 Message-ID: <86zgd2pyrw.wl-maz@kernel.org> From: Marc Zyngier To: Peter Xu Cc: Gavin Shan , kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, shuah@kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, ajones@ventanamicro.com, bgardon@google.com, dmatlack@google.com, will@kernel.org, suzuki.poulose@arm.com, alexandru.elisei@arm.com, pbonzini@redhat.com, seanjc@google.com, oliver.upton@linux.dev, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: Re: [PATCH v8 3/7] KVM: Support dirty ring in conjunction with bitmap In-Reply-To: References: <20221104234049.25103-1-gshan@redhat.com> <20221104234049.25103-4-gshan@redhat.com> <87o7tkf5re.wl-maz@kernel.org> <87iljrg7vd.wl-maz@kernel.org> <867d07qfvk.wl-maz@kernel.org> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (aarch64-unknown-linux-gnu) MULE/6.0 (HANACHIRUSATO) Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: peterx@redhat.com, gshan@redhat.com, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, shuah@kernel.org, catalin.marinas@arm.com, andrew.jones@linux.dev, ajones@ventanamicro.com, bgardon@google.com, dmatlack@google.com, will@kernel.org, suzuki.poulose@arm.com, alexandru.elisei@arm.com, pbonzini@redhat.com, seanjc@google.com, oliver.upton@linux.dev, zhenyzha@redhat.com, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Message-ID: <20221107153059.cm7q0dLCLP0jSj1uEm1UPs-3vlCCqfF3Bzz-_FP58Ts@z> On Mon, 07 Nov 2022 14:59:41 +0000, Peter Xu wrote: > > On Mon, Nov 07, 2022 at 09:21:35AM +0000, Marc Zyngier wrote: > > On Sun, 06 Nov 2022 21:06:43 +0000, > > Peter Xu wrote: > > > > > > It's definitely not the case for x86, but if it's true for > > > arm64, then could the DMA be spread across all the guest pages? > > > If it's also true, I really don't know how this will work.. > > > > Of course, all pages can be the target of DMA. It works the same way > > it works for the ITS: you sync the state, you obtain the dirty bits, > > you move on. > > > > And mimicking what x86 does is really not my concern (if you still > > think that arm64 is just another flavour of x86, stay tuned! ;-). > > I didn't mean so, I should probably stop mentioning x86. :) Please! I turned off my last x86 development machine over the weekend, and my x86 laptop is now a glorified window manager... ;-) > I had some sense already from the topics in past few years of kvm forum. > Yeah I'll be looking forward to anything more coming. Yup. Hopefully we won't have to wait for too long to see this stuff (I had good discussions on the subject at both KF and Plumbers in Dublin earlier this year). > > > We're only syncing the dirty bitmap once right now with the > > > protocol. If that can cover most of the guest mem, it's same as > > > non-live. If we sync it periodically, then it's the same as > > > enabling dirty-log alone and the rings are useless. > > > > I'm glad that you finally accept it: the ring *ARE* useless in the > > general sense. Only limited, CPU-only workloads can make any use of > > the current design. This probably covers a large proportion of what > > the cloud vendors do, but this doesn't work for general situations > > where you have a stream of dirty pages originating outside of the > > CPUs. > > The ring itself is really not the thing to blame, IMHO it's a good attempt > to try out de-coupling guest size in regard of dirty tracking from kvm. It > may not be perfect, but it may still service some of the goals, e.g., at > least it allows the user app to detect per-vcpu information and also since > there's the ring full events we can do something more than before like the > vcpu throttling that China Telecom does with the ring structures. I don't disagree with that: for vcpu-based workloads, the rings are great and doing their job. It's just that there is another side to this problem, and you'll have to deal with both eventually. We're just ahead of the curve here... Thanks, M. -- Without deviation from the norm, progress is not possible.