From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9C23DC433F5 for ; Sun, 13 Feb 2022 10:25:20 +0000 (UTC) Received: from localhost ([::1]:51444 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1nJC4F-0000Wp-3Q for qemu-devel@archiver.kernel.org; Sun, 13 Feb 2022 05:25:19 -0500 Received: from eggs.gnu.org ([209.51.188.92]:55168) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJC27-0008Fs-M1 for qemu-devel@nongnu.org; Sun, 13 Feb 2022 05:23:07 -0500 Received: from dfw.source.kernel.org ([139.178.84.217]:58834) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1nJC25-0005ZN-35 for qemu-devel@nongnu.org; Sun, 13 Feb 2022 05:23:07 -0500 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 80B6F60E9D; Sun, 13 Feb 2022 10:22:56 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id E7ED5C004E1; Sun, 13 Feb 2022 10:22:55 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1644747775; bh=qVPXummkbgn66HB7C9DZJnL9F7+cvb3dt9lBQTxDi7w=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=B1TFU1LLS2aenZT4B3QimSep5+igrgjdm3tdaUlIhoobOPGQ321QimM+Peq1u5L10 xEz6c4p4lMaqOXm0pGnGgZN4oCJm2MG1e2thS9CWjMzulWd4gOTlBbsfPC9+6TONdH PCqpnQUjetA86kwKcykmLVs3ZqBDzTPOoQgiOO/A41G0m/Qqccsnm57tHAihqyYhny O12/tTzM0Chd2pOrs3pxYWt/TpJuiPS+cYTmkDSL17+CilthH25PXL8a2AfEZj/86r ZB1AgLoKpEotiH0XqCmcpCdrjyocLWf4MpBNkUUEvizkujNhqBR0VdIuwM4tOd+jgH E6+bcGju4Y60A== Received: from sofa.misterjones.org ([185.219.108.64] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.94.2) (envelope-from ) id 1nJC1t-007THY-Ko; Sun, 13 Feb 2022 10:22:53 +0000 Date: Sun, 13 Feb 2022 10:22:53 +0000 Message-ID: <87iltjxdz6.wl-maz@kernel.org> From: Marc Zyngier To: Akihiko Odaki Subject: Re: [PULL 18/38] hw/arm/virt: Honor highmem setting when computing the memory map In-Reply-To: <3f4f5e98-fcb8-bf4d-e464-6ad365af92f8@gmail.com> References: <20220120123630.267975-1-peter.maydell@linaro.org> <20220120123630.267975-19-peter.maydell@linaro.org> <3f4f5e98-fcb8-bf4d-e464-6ad365af92f8@gmail.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.219.108.64 X-SA-Exim-Rcpt-To: akihiko.odaki@gmail.com, qemu-devel@nongnu.org, peter.maydell@linaro.org, drjones@redhat.com, eric.auger@redhat.com, kvmarm@lists.cs.columbia.edu, kvm@vger.kernel.org, kernel-team@android.com, agraf@csgraf.de X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Received-SPF: pass client-ip=139.178.84.217; envelope-from=maz@kernel.org; helo=dfw.source.kernel.org X-Spam_score_int: -70 X-Spam_score: -7.1 X-Spam_bar: ------- X-Spam_report: (-7.1 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Andrew Jones , kvm@vger.kernel.org, qemu-devel@nongnu.org, Eric Auger , Alexander Graf , kernel-team@android.com, kvmarm@lists.cs.columbia.edu Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" [+ Alex for HVF] On Sun, 13 Feb 2022 05:05:33 +0000, Akihiko Odaki wrote: > > On 2022/01/20 21:36, Peter Maydell wrote: > > From: Marc Zyngier > > > > Even when the VM is configured with highmem=off, the highest_gpa > > field includes devices that are above the 4GiB limit. > > Similarily, nothing seem to check that the memory is within > > the limit set by the highmem=off option. > > > > This leads to failures in virt_kvm_type() on systems that have > > a crippled IPA range, as the reported IPA space is larger than > > what it should be. > > > > Instead, honor the user-specified limit to only use the devices > > at the lowest end of the spectrum, and fail if we have memory > > crossing the 4GiB limit. > > > > Reviewed-by: Andrew Jones > > Reviewed-by: Eric Auger > > Signed-off-by: Marc Zyngier > > Message-id: 20220114140741.1358263-4-maz@kernel.org > > Signed-off-by: Peter Maydell > > --- > > hw/arm/virt.c | 10 +++++++--- > > 1 file changed, 7 insertions(+), 3 deletions(-) > > > > diff --git a/hw/arm/virt.c b/hw/arm/virt.c > > index 62bdce1eb4b..3b839ba78ba 100644 > > --- a/hw/arm/virt.c > > +++ b/hw/arm/virt.c > > @@ -1670,7 +1670,7 @@ static uint64_t virt_cpu_mp_affinity(VirtMachineState *vms, int idx) > > static void virt_set_memmap(VirtMachineState *vms) > > { > > MachineState *ms = MACHINE(vms); > > - hwaddr base, device_memory_base, device_memory_size; > > + hwaddr base, device_memory_base, device_memory_size, memtop; > > int i; > > vms->memmap = extended_memmap; > > @@ -1697,7 +1697,11 @@ static void virt_set_memmap(VirtMachineState *vms) > > device_memory_size = ms->maxram_size - ms->ram_size + ms->ram_slots * GiB; > > /* Base address of the high IO region */ > > - base = device_memory_base + ROUND_UP(device_memory_size, GiB); > > + memtop = base = device_memory_base + ROUND_UP(device_memory_size, GiB); > > + if (!vms->highmem && memtop > 4 * GiB) { > > + error_report("highmem=off, but memory crosses the 4GiB limit\n"); > > + exit(EXIT_FAILURE); > > + } > > if (base < device_memory_base) { > > error_report("maxmem/slots too huge"); > > exit(EXIT_FAILURE); > > @@ -1714,7 +1718,7 @@ static void virt_set_memmap(VirtMachineState *vms) > > vms->memmap[i].size = size; > > base += size; > > } > > - vms->highest_gpa = base - 1; > > + vms->highest_gpa = (vms->highmem ? base : memtop) - 1; > > if (device_memory_size > 0) { > > ms->device_memory = g_malloc0(sizeof(*ms->device_memory)); > > ms->device_memory->base = device_memory_base; > > Hi, > This breaks in a case where highmem is disabled but can have more than > 4 GiB of RAM. M1 (Apple Silicon) actually can have 36-bit PA with HVF, > which is not enough for highmem MMIO but is enough to contain 32 GiB > of RAM. Funny. The whole point of this series is to make it all work correctly on M1. > Where the magic number of 4 GiB / 32-bit came from? Not exactly a magic number. From QEMU's docs/system/arm/virt.rst: highmem Set ``on``/``off`` to enable/disable placing devices and RAM in physical address space above 32 bits. The default is ``on`` for machine types later than ``virt-2.12``. TL;DR: Removing the bogus 'highmem=off' option from your command-line should get you going with large memory spaces, up to the IPA limit. The fact that you could run with 32GB of RAM while mandating that the guest IPA space was limited to 32bit was nothing but a bug, further "exploited" by HVF to allow disabling the highhmem devices which are out of reach given the HW limitations (see [1] for details on the discussion, specially around patch 3). This is now fixed, and has been extended to work with any IPA size (including 36bit machines such as M1). > I also don't quite understand what failures virt_kvm_type() had. QEMU works by first computing the memory map and passing the required IPA limit to KVM as part of the VM type. By failing to take into account the initial limit requirements to the IPA space (either via a command-line option such as 'highmem', or by using the value provided by KVM itself), QEMU would try to create a VM that cannot run on the HW, and KVM would simply return an error. All of this is documented as part of the KVM/arm64 API [2]. And with this fixed, QEMU is able to correctly drive KVM on M1. M. [1] https://lore.kernel.org/all/20210822144441.1290891-1-maz@kernel.org [2] https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/Documentation/virt/kvm/api.rst#n138 -- Without deviation from the norm, progress is not possible.