From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 2002:a17:906:3b11:0:0:0:0 with SMTP id g17csp356180ejf; Thu, 11 Aug 2022 00:41:05 -0700 (PDT) X-Google-Smtp-Source: AA6agR64Mng6tn/BYKzPXyOgV9DfjYY5u5U58VO2qql1dfM6++8Rx4xEdkvTM8rWxQHTLiXIOIzB X-Received: by 2002:a05:620a:899:b0:6b9:73e0:ca46 with SMTP id b25-20020a05620a089900b006b973e0ca46mr10035041qka.397.1660203665100; Thu, 11 Aug 2022 00:41:05 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1660203665; cv=none; d=google.com; s=arc-20160816; b=KdzuiOPRsUTnd+TwuTd2jEkIyBeXd7qpLtxcUohdinBeCBbGnEOGvwSpfFPxE/fKE3 vfslGIgBNHXDBb/biykbwwiX4bJm9rKk6ES9XEfhmr0IODWONwJLh8XKmqWHD0lvIT8d VIBDqYzvQI6+g/chUREULpjSdV0CWBzE4tHmuOSxeojYliKk40/oXrQvg7y6ikJsMJtM 95FbJCDpinIWs6Mmyedur5DhkI3n/6ocSffHBM8BpRNlsFcJMbb7DENrH94wIWpUIC8W vdlr2KGsA9EywRwZj3E2kISQGZKDxxt2CSemJoHFd6g0qWt/VPRfTqng+rW4puoSgX39 KNIQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:mime-version:user-agent :references:in-reply-to:subject:cc:to:from:message-id:date :dkim-signature; bh=VDvDIty+fDdPoE6AGYQV7K8H5GnClOzG+8fFbzcZm24=; b=OPXnsFtabF6TYTnNAtUlNOJS0pJX8C7ylrdU3iOL6LOGJW7Q2F85LAKEB6GrTY/pCr OgmzSDfyad58W63hvjmgf9T/DcAI761FPlOYO3yS4eKHv9cNVQZwL2wFMe/fmmqq9BU5 MVer93ac6XsmksLpq70ubgy6hH+rq0wMQHq8WXOuFut1e5Gf+lou5TFS26twjV71A9ul S14ETgD40Yf0Q4dQ7HrtHEjgZeVd6RyntS9cQKiPkcO+m2Lhyj1/Oi9p8uKTGT/wsXBy GxvgE8Eb8h77+Q/dQnVLUK/ErF+5Gr3RfgzeqZwSDuU60bWl0GlcyExujSv2ggolqE/c 6LBw== ARC-Authentication-Results: i=1; mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rVlSNlEk; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Return-Path: Received: from lists.gnu.org (lists.gnu.org. [209.51.188.17]) by mx.google.com with ESMTPS id dl8-20020a05620a1d0800b006b8da816509si1138087qkb.257.2022.08.11.00.41.04 for (version=TLS1_2 cipher=ECDHE-ECDSA-CHACHA20-POLY1305 bits=256/256); Thu, 11 Aug 2022 00:41:05 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) client-ip=209.51.188.17; Authentication-Results: mx.google.com; dkim=pass header.i=@kernel.org header.s=k20201202 header.b=rVlSNlEk; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 209.51.188.17 as permitted sender) smtp.mailfrom="qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org"; dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=kernel.org Received: from localhost ([::1]:46628 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1oM2oS-0004ln-Ia for alex.bennee@linaro.org; Thu, 11 Aug 2022 03:41:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:53360) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oM2lZ-0002gR-GI; Thu, 11 Aug 2022 03:38:05 -0400 Received: from ams.source.kernel.org ([145.40.68.75]:42190) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1oM2lU-0003bP-9i; Thu, 11 Aug 2022 03:38:05 -0400 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by ams.source.kernel.org (Postfix) with ESMTPS id 588D6B81E64; Thu, 11 Aug 2022 07:37:51 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 1B880C433D6; Thu, 11 Aug 2022 07:37:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1660203470; bh=DZ8dklJXQFgZlGroTGSSTZwoulndxwpHNk1hcxTh7Fw=; h=Date:From:To:Cc:Subject:In-Reply-To:References:From; b=rVlSNlEkoHweETS/0xcxpV3bKDfMeAxFYXkAqdwbO/c+pvuohrDKGmnGcVOrevIP7 IMys3KMoyEMckKPPCwtJKY/F1irWJ488i7gcp+8szvFaVuOoVFfnWaOjx5KRuDXirK efY1wz+Qaviwghh5wT4NIz1cqtMlKCbLKk2YkTCkos6e3bgoGy6ifRpnf+KU65NK9x vMsHzi5DkTOOgssU2t8FjFxcxvoqWelIR5m9jTaNjoGwM4OBbhl9uFBuM3OnFCkiZH 2VRUde0dD86zZ/mL7fwo5YjmCuOaMG6y9xko78wr7nzigxy46OFET2TPThKLyUIV9x z4ZOj/Yx+taoQ== Received: from ip-185-104-136-29.ptr.icomera.net ([185.104.136.29] helo=wait-a-minute.misterjones.org) by disco-boy.misterjones.org with esmtpsa (TLS1.3) tls TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 (Exim 4.95) (envelope-from ) id 1oM2lH-002I5o-Pb; Thu, 11 Aug 2022 08:37:47 +0100 Date: Thu, 11 Aug 2022 08:37:46 +0100 Message-ID: <87k07fb585.wl-maz@kernel.org> From: Marc Zyngier To: Gavin Shan Cc: eric.auger@redhat.com, qemu-arm@nongnu.org, qemu-devel@nongnu.org, peter.maydell@linaro.org, richard.henderson@linaro.org, cohuck@redhat.com, zhenyzha@redhat.com, shan.gavin@gmail.com Subject: Re: [PATCH 1/2] hw/arm/virt: Improve address assignment for highmem IO regions In-Reply-To: <6669dd3f-c1cd-e3f3-21a7-afab4deb91f5@redhat.com> References: <20220802064529.547361-1-gshan@redhat.com> <20220802064529.547361-2-gshan@redhat.com> <9c8365c6-d27b-df76-371d-bd32ca2a26f7@redhat.com> <87tu6tbyk9.wl-maz@kernel.org> <0ed2ebc7-8d6e-7555-3af4-31eb071a584b@redhat.com> <87fsi7w0vn.wl-maz@kernel.org> <6669dd3f-c1cd-e3f3-21a7-afab4deb91f5@redhat.com> User-Agent: Wanderlust/2.15.9 (Almost Unreal) SEMI-EPG/1.14.7 (Harue) FLIM-LB/1.14.9 (=?UTF-8?B?R29qxY0=?=) APEL-LB/10.8 EasyPG/1.0.0 Emacs/27.1 (x86_64-pc-linux-gnu) MULE/6.0 (HANACHIRUSATO) MIME-Version: 1.0 (generated by SEMI-EPG 1.14.7 - "Harue") Content-Type: text/plain; charset=US-ASCII X-SA-Exim-Connect-IP: 185.104.136.29 X-SA-Exim-Rcpt-To: gshan@redhat.com, eric.auger@redhat.com, qemu-arm@nongnu.org, qemu-devel@nongnu.org, peter.maydell@linaro.org, richard.henderson@linaro.org, cohuck@redhat.com, zhenyzha@redhat.com, shan.gavin@gmail.com X-SA-Exim-Mail-From: maz@kernel.org X-SA-Exim-Scanned: No (on disco-boy.misterjones.org); SAEximRunCond expanded to false Received-SPF: pass client-ip=145.40.68.75; envelope-from=maz@kernel.org; helo=ams.source.kernel.org X-Spam_score_int: -71 X-Spam_score: -7.2 X-Spam_bar: ------- X-Spam_report: (-7.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.082, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_HI=-5, SPF_HELO_NONE=0.001, T_SCC_BODY_TEXT_LINE=-0.01, T_SPF_TEMPERROR=0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-arm@nongnu.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org Sender: "Qemu-arm" X-TUID: noxoWxB1guff Hi Gavin, On Thu, 11 Aug 2022 06:32:36 +0100, Gavin Shan wrote: > > Hi Marc, > > On 8/8/22 7:17 PM, Marc Zyngier wrote: > > On Wed, 03 Aug 2022 14:02:04 +0100, > > Gavin Shan wrote: [...] > Sorry for the delay. I think the original changelog is confusing > enough and sorry about it. I will improve it if v2 is needed :) > > Yes, we shouldn't assign address to VIRT_HIGH_PCIE_ECAM region > and enable it when vms->highmem_ecam is false in virt_set_memmap(). > > In the original implementation of virt_set_memmap(), the memory > regioin is disabled when when vms->highmem_ecam is false. However, > the address is still assigned to the memory region, even when > vms->highmem_ecam is false. This leads to waste in the PA space. Yes, I got this point now. However, I think the approach you've chosen leads to subtle issues, see below. [...] > >> Eric thought that VIRT_HIGH_GIC_REDIST2 and VIRT_HIGH_PCIE_MMIO regions > >> aren't user selectable. I tended to explain why it's not true. 'maxmem=' > >> can affect the outcome. When 'maxmem=' value is big enough, there will be > >> no free area in the PA space to hold those two regions. > > > > Right, that's an interesting point. This is a consequence of these > > upper regions floating above RAM. > > > > Yep, it's fine for those high memory region floating above RAM, and to > disable them if we run out of PA space. Something may be irrelevant > to this topic: VIRT_HIGH_PCIE_MMIO region has 512GB, which is huge one. > It may be nice to fall back smaller sizes when we're having tight PA > space. For example, we can fall back to try 256GB if 512GB doesn't fit > into the PA space. > > However, I'm not sure how we had the size (512GB) for the region. Are > there practical factors why we need a 512GB PCIe 64-bits MMIO space? I have no idea. But this is something that is now relied upon by existing VMs, and I'm not sure we can break this. > >>>>>> Improve address assignment for highmem IO regions to avoid the waste > >>>>>> in the PA space by putting the logic into virt_memmap_fits(). > >>> > >>> I guess that this is what I understand the least. What do you mean by > >>> "wasted PA space"? Either the regions fit in the PA space, and > >>> computing their addresses in relevant, or they fall outside of it and > >>> what we stick in memap[index].base is completely irrelevant. > >>> > >> > >> It's possible that we run into the following combination. we should > >> have enough PA space to enable VIRT_HIGH_PCIE_MMIO region. However, > >> the region is disabled in the original implementation because > >> VIRT_HIGH_{GIC_REDIST2, PCIE_ECAM} regions consumed 1GB, which is > >> unecessary and waste in the PA space. > >> > >> static MemMapEntry extended_memmap[] = { > >> [VIRT_HIGH_GIC_REDIST2] = { 0x0, 64 * MiB }, > >> [VIRT_HIGH_PCIE_ECAM] = { 0x0, 256 * MiB }, > >> [VIRT_HIGH_PCIE_MMIO] = { 0x0, 512 * GiB }, > >> }; > >> > >> IPA_LIMIT = (1UL << 40) > >> '-maxmem' = 511GB /* Memory starts from 1GB */ > >> '-slots' = 0 > >> vms->highmem_rdist2 = false > >> vms->highmem_ecam = false > >> vms->highmem_mmio = true > >> > > > > Sure. But that's not how QEMU works today, and these regions are > > enabled in order (though it could well be that my recent changes have > > made the situation more complicated). > > > > What you're suggesting is a pretty radical change in the way the > > memory map is set. My hunch is that allowing the highmem IO regions to > > be selectively enabled and allowed to float in the high IO space > > should come as a new virt machine revision, with user-visible options. > > > > Yeah, These regions are enabled in order. It also means they have ascending > priorities. In other words, '-maxmem' has higher priority than those 3 high > memory regions. > > My suggested code changes just improve the address assignment for these 3 > high memory regions, without changing the mechanism fundamentally. The > intention of the proposed changes are like below. > > - In virt_set_memmap(), don't assign address for one specific high memory > region if it has been disabled. > > - Put the logic into standalone helper, to simplify the code. > > I'm not sure about the user-visible options. I would say it's going to > increase user's load. I means the user experience will be degraded. > Those user-visible options needs to be worried by users :) Not necessarily. You could have a default that picks whatever fits. But this is hard to get right, and I'm not convinced you will always be able to satisfy everyone. I'm also worried of seeing different behaviours if I dump a VM from QEMU 7.x and reload it with 7.y, only to realise that I don't have the same memory layout. In my opinion, these things need to be either constant, or user-specified. > Marc, lets improve the changelog and the code changes in v2, to seek > your comments if you agree? :) Of course! M. -- Without deviation from the norm, progress is not possible.