From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 59E5EC3DA7D for ; Tue, 3 Jan 2023 17:21:33 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:To:From:Date:Reply-To:Cc:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/+CJfySzK/ciQPMr+jx/rEwf/2yWJMUyq0o/vTdY7p4=; b=2TxCHkxsaZsUAT E/3Suzkz6m3GpgreoIfK4iTwOicOOGFFSdbZ3GYwPG0epV+3m51j2rgYqWHoL69QnmND+JoErGRPV yysYLjNKG4uYt9y0pZ/qSxuCXL7KLyWf3s5TyCzPrIYPBDHzD8xc3WqsZFVrwnsd9E0KAXRJpS81C r9uBH9Q4Fqa7Qwu69R+drI/SaMPI0XxN9jTGk04ED/gmFRPBz+Qqd4L+NVAkEuGKE2pbkyrtPwAvM yUpOqDhk+HXXfuIkHb9zyVOvqRhzvyJ8KwtLaiHW8qTcigAw1EZpcx4ilH60aZFQNYawi1sj7qANi OcktC1U1McS+p0JjrvWw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pCkxK-003LDL-PB; Tue, 03 Jan 2023 17:20:07 +0000 Received: from foss.arm.com ([217.140.110.172]) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pCiGs-001xqG-5S for linux-arm-kernel@lists.infradead.org; Tue, 03 Jan 2023 14:28:08 +0000 Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 98DDE152B; Tue, 3 Jan 2023 06:28:45 -0800 (PST) Received: from monolith.localdoman (unknown [172.31.20.19]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id 617043F587; Tue, 3 Jan 2023 06:28:02 -0800 (PST) Date: Tue, 3 Jan 2023 14:27:59 +0000 From: Alexandru Elisei To: will@kernel.org, mark.rutland@arm.com, linux-arm-kernel@lists.infradead.org, maz@kernel.org, james.morse@arm.com, suzuki.poulose@arm.com, oliver.upton@linux.dev, kvmarm@lists.linux.dev, kvmarm@lists.cs.columbia.edu Subject: Re: KVM: arm64: A new approach for SPE support Message-ID: References: MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230103_062806_384351_664ABBCF X-CRM114-Status: GOOD ( 41.03 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org Hi, Gentle ping regarding this. Thanks, Alex On Wed, Nov 23, 2022 at 11:40:45AM +0000, Alexandru Elisei wrote: > The previous discussion about how best to add SPE support to KVM [1] is > heading in the direction of pinning at EL2 only the buffer, when the guest > enables profiling, instead of pinning the entire VM memory. Although better > than pinning the entire VM at EL2, it still has some disadvantages: > > 1. Pinning memory at stage 2 goes against the design principle of secondary > MMUs, which must reflect all changes in the primary (host's stage 1) page > tables. This means a mechanism by which to pin VM memory at stage 2 must be > created from scratch just for SPE. Although I haven't done this yet, I'm a > bit concerned that this will turn out to be fragile and/or complicated. > > 2. The architecture allows software to change the VA to IPA translations > for the profiling buffer when the buffer is enabled if profiling is > disabled (the buffer is enabled, but sampling is disabled). Since SPE can > be programmed to profile EL0 only, and there is no easy way for KVM to trap > the exact moment when profiling becomes enabled in this scenario to > translate the buffer's guest VAs to IPA, to pin the IPAs at stage 2, it is > required for KVM impose limitations on how a guest uses SPE for emulation > to work. > > I've prototyped a new approach [2] which eliminates both disadvantages, but > comes with its own set of drawbacks. The approach I've been working on is > to have KVM allocate a buffer in the kernel address space to profile the > guest, and when the buffer becomes full (or profiling is disabled for other > reasons), to copy the contents of the buffer to guest memory. > > I'll start with the advantages: > > 1. No memory pinning at stage 2. > > 2. No meaningful restrictions on how the guest programs SPE, since the > translation of the guest VAs to IPAs is done by KVM when profiling has been > completed. > > 3. Neoverse N1 errata 1978083 ("Incorrect programming of PMBPTR_EL1 might > result in a deadlock") [6] is handled without any extra work. > > As I see it, there are two main disadvantages: > > 1. The contents of the KVM buffer must be copied to the guest. In the > prototype this is done all at once, when profiling is stopped [3]. > Presumably this can be amortized by unmapping the pages corresponding to > the guest buffer from stage 2 (or marking them as invalid) and copying the > data when the guest reads from those pages. Needs investigating. > > 2. When KVM profiles the guest, the KVM buffer owning exception level must > necessarily be EL2. This means that while profiling is happening, > PMBIDR_EL1.P = 1 (programming of the buffer is not allowed). PMBIDR_EL1 > cannot be trapped without FEAT_FGT, so a guest that reads the register > after profiling becomes enabled will read the P bit as 1. I cannot think of > any valid reason for a guest to look at the bit after enabling profiling. > With FEAT_FGT, KVM would be able to trap accesses to the register. > > 3. In the worst case scenario, when the entire VM memory is mapped in the > host, this approach consumes more memory because the memory for the buffer > is separate from the memory allocated to the VM. On the plus side, there > will always be less memory pinned in the host for the VM process, since > only the buffer has to be pinned, instead of the buffer plus the guest's > stage 1 translation tables (to avoid SPE encountering a stage 2 fault on a > stage 1 translation table walk). Could be mitigated by providing an ioctl > to userspace to set the maximum size for the buffer. > > I prefer this new approach instead of pinning the buffer at stage 2. It is > straightforward, less fragile and doesn't limit how a guest can program > SPE. > > As for the prototype, I wrote it as a quick way to check if this approach > is viable. Does not have SPE support for the nVHE case because I would have > had to figure out how to map a continuous VA range in the EL2's translation > tables; supporting only the VHE case was a lot easier. The prototype > doesn't have a stage 1 walker, so it's limited to guests that use id-mapped > addresses from TTBR0_EL1 for the buffer (although it would be trivial to > modify it to accept addresses from TTBR1_EL1) - I've used kvm-unit-tests > for testing [4]. I've tested the prototype on the model and on an Ampere > Altra. > > For those interested, kvmtool support to run the prototype has also been > added [5] (add --spe to the command line to run a VM). > > [1] https://lore.kernel.org/all/Yl6+JWaP+mq2Nc0b@monolith.localdoman/ > [2] https://gitlab.arm.com/linux-arm/linux-ae/-/tree/kvm-spe-v6-copy-buffer-wip4-without-nvhe > [3] https://gitlab.arm.com/linux-arm/linux-ae/-/blob/kvm-spe-v6-copy-buffer-wip4-without-nvhe/arch/arm64/kvm/spe.c#L197 > [4] https://gitlab.arm.com/linux-arm/kvm-unit-tests-ae/-/tree/kvm-spe-v6-copy-buffer-wip4 > [5] https://gitlab.arm.com/linux-arm/kvmtool-ae/-/tree/kvm-spe-v6-copy-buffer-wip4 > [6] https://developer.arm.com/documentation/SDEN885747/latest > > Thanks, > Alex > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel