From mboxrd@z Thu Jan 1 00:00:00 1970 Received: by 2002:adf:b64b:0:0:0:0:0 with SMTP id i11-v6csp2230151wre; Thu, 24 May 2018 12:26:56 -0700 (PDT) X-Google-Smtp-Source: ADUXVKJ0EM2rPKs/wtMtWmn/fOb9wFAlDfyLrMfCE2TBU9HLDMIAeKtsKd6AZhWAd8tKSo387Lgh X-Received: by 2002:ac8:27c9:: with SMTP id x9-v6mr8453761qtx.374.1527190016101; Thu, 24 May 2018 12:26:56 -0700 (PDT) ARC-Seal: i=1; a=rsa-sha256; t=1527190016; cv=none; d=google.com; s=arc-20160816; b=RHlN5W2l0FY0waw+kiLKj5nLlb8MjBvmqq0Bk9cjFN0kG7QPHYsJdSIyEecQeO9FQ/ cYtu8QxTCseUzgJWxLMSq3NHyC1kdARLcHBDXlJbcd6kqhhjB81ewU8uHZ0FjTDbWkaB I/7+ABIqvsFcG/F2tqboQz/zRfhGaUSJPUTlScVKU2Mmolh6FCaA5cZNYVNz50hK1fAC 2G8yTmi7GdLrZc2NzWgN3tQhGjqlyzINYanCuhy1jPnSWKerdS3/yiFKqBb14ONKoLY6 q+jb5JLBOUA3kaTnfaRAwQlbStcWRAyzXYapkt3q/vqoB9+a735JVr8UMbFk5TQKbSEf 50Vg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816; h=sender:errors-to:cc:list-subscribe:list-help:list-post:list-archive :list-unsubscribe:list-id:precedence:subject :content-transfer-encoding:in-reply-to:mime-version:user-agent:date :message-id:from:references:to:arc-authentication-results; bh=uC4gNpNxxDvBDCk8qj/WHRGlbtDB27nsxgOF4ormpBs=; b=rrm5hqYZQaiTd/2x4hvKXtqYjT3oTJOW73wFCRgkXjQSV5lloST1K9qsdJ9j3Bjj0n bdc7oSVuQudpikxBmr+DSGavOk5n7DRCQEVWtQAGLtaJVIQk5AdBd7R7/BZGYgg8Ja33 13WaiQJ2dT6+XayXVzWrWC5i2L4GpyycM2EkILRFpAeSj4Sci8Zw89FnpE874dFD+d8J c7B7QhJG9OnFqSZwdCnsBpwcwKI+YRGgbDN6O2Oi3FC2vsriNDcJL3LFnjTSoHxh1ln0 B+PS9BKC0LTfy/5t5QfSsInF34CjuaMNOgLvs62M5f9Yc2cObA2spxDVilw5GhkdWHpP COOA== ARC-Authentication-Results: i=1; mx.google.com; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Return-Path: Received: from lists.gnu.org (lists.gnu.org. [2001:4830:134:3::11]) by mx.google.com with ESMTPS id q11-v6si6374022qvh.26.2018.05.24.12.26.55 for (version=TLS1 cipher=AES128-SHA bits=128/128); Thu, 24 May 2018 12:26:56 -0700 (PDT) Received-SPF: pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) client-ip=2001:4830:134:3::11; Authentication-Results: mx.google.com; spf=pass (google.com: domain of qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org designates 2001:4830:134:3::11 as permitted sender) smtp.mailfrom=qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org; dmarc=fail (p=NONE sp=NONE dis=NONE) header.from=redhat.com Received: from localhost ([::1]:40404 helo=lists.gnu.org) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLvt1-0008PA-JP for alex.bennee@linaro.org; Thu, 24 May 2018 15:26:55 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49946) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLvso-0008Nm-RW for qemu-arm@nongnu.org; Thu, 24 May 2018 15:26:44 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLvsm-0008Ge-5y for qemu-arm@nongnu.org; Thu, 24 May 2018 15:26:42 -0400 Received: from mx3-rdu2.redhat.com ([66.187.233.73]:60258 helo=mx1.redhat.com) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1fLvsl-0008GM-W9; Thu, 24 May 2018 15:26:40 -0400 Received: from smtp.corp.redhat.com (int-mx03.intmail.prod.int.rdu2.redhat.com [10.11.54.3]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1.redhat.com (Postfix) with ESMTPS id 5E8204000786; Thu, 24 May 2018 19:26:39 +0000 (UTC) Received: from localhost.localdomain (ovpn-116-69.ams2.redhat.com [10.36.116.69]) by smtp.corp.redhat.com (Postfix) with ESMTPS id D0ACB1006EBE; Thu, 24 May 2018 19:26:33 +0000 (UTC) To: Laszlo Ersek , Ard Biesheuvel References: <1527091418-11874-1-git-send-email-eric.auger@redhat.com> <22c4e504-a7b4-e6dd-b2cc-618d306b6f0c@redhat.com> <550464ac-5155-6311-d0f7-92c7c0813f82@redhat.com> From: Auger Eric Message-ID: Date: Thu, 24 May 2018 21:26:32 +0200 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Scanned-By: MIMEDefang 2.78 on 10.11.54.3 X-Greylist: Sender IP whitelisted, not delayed by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Thu, 24 May 2018 19:26:39 +0000 (UTC) X-Greylist: inspected by milter-greylist-4.5.16 (mx1.redhat.com [10.11.55.6]); Thu, 24 May 2018 19:26:39 +0000 (UTC) for IP:'10.11.54.3' DOMAIN:'int-mx03.intmail.prod.int.rdu2.redhat.com' HELO:'smtp.corp.redhat.com' FROM:'eric.auger@redhat.com' RCPT:'' X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] [fuzzy] X-Received-From: 66.187.233.73 Subject: Re: [Qemu-arm] [Qemu-devel] [RFC 0/2] ARM virt: Support up to 256 PCIe buses X-BeenThere: qemu-arm@nongnu.org X-Mailman-Version: 2.1.21 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Peter Maydell , Andrew Jones , QEMU Developers , qemu-arm , Shannon Zhao , Eric Auger Errors-To: qemu-arm-bounces+alex.bennee=linaro.org@nongnu.org Sender: "Qemu-arm" X-TUID: a3+XWcipCwcb On 05/24/2018 07:20 PM, Laszlo Ersek wrote: > On 05/24/18 16:14, Ard Biesheuvel wrote: >> On 24 May 2018 at 15:59, Laszlo Ersek wrote: >>> On 05/24/18 15:07, Peter Maydell wrote: >>>> On 24 May 2018 at 13:59, Laszlo Ersek wrote: >>>>> On 05/24/18 11:11, Peter Maydell wrote: >>>>>> Won't it also break a guest which is just Linux loaded not via >>>>>> firmware which is an aarch32 kernel without LPAE support? >>>>> >>>>> Does such a thing exist? (I honestly have no clue.) >>>> >>>> Yes, it does; LPAE isn't a mandatory kernel config option. >>>> This is why we have the machine 'highmem' option, so that >>>> we can run on those kernels by not putting anything above >>>> the 4G boundary. Looking back at the history on that, we >>>> opted at the time for "default to highmem on, and if you're >>>> running an non-lpae kernel you need to turn it off manually". >>> >>> Ah, OK, I didn't know that. >>> >>>> So we can handle those kernels by just not putting ECAM >>>> above 4G if highmem is false. >>> >>> The problem is we can have a combination of 32-bit UEFI firmware (which >>> certainly lacks LPAE) and a 32-bit kernel which supports LPAE. >>> Previously, you wouldn't specify highmem=off, and things would just work >>> -- the firmware would simply ignore the >=4GB MMIO aperture, and use the >>> 32-bit MMIO aperture only (and use the sole 32-bit ECAM). The kernel >>> could then use both low and high MMIO apertures, however (I gather?). >>> >>> The difference with "high ECAM" is that it is *moved* (not *added*), so >>> the 32-bit firmware is left with nothing for config space access. For >>> booting the same combination as above, you are suddenly forced to add >>> highmem=off, just to keep the ECAM low -- and that, while it keeps the >>> firmware happy, prevents the LPAE-capable kernel from using the high >>> MMIO aperture. >>> >>> So I think "highmem_ecam" should be computed like this: >>> >>> highmem_ecam = highmem_ecam_machtype_default && >>> highmem && >>> (!firmware_loaded || aarch64); >>> >> >> Given that the firmware is tightly coupled to the platform, we may >> decide not to care about ECAM for UEFI itself, and invent a secondary >> config space access mechanism that does not consume such a huge amount >> of address space. For instance, legacy PCI uses a pair of I/O ports >> for this, one to set the address and one to perform the actual read or >> write, and we could easily implement something similar (such an >> interface is problematic in SMP context but we don't care about that >> anyway) >> >> Just a thought - perhaps we don't care enough about 32-bit to go >> through the trouble, but it would be nice if LPAE capable 32-bit >> guests could make use of the expanded PCIe config space as well. > > Under the above proposal, they could, they'd just have to be launched > without firmware: > > highmem_ecam_machtype_default = true; > highmem = true; > firmware_loaded = false; > aarch64 = false; > > highmem_ecam = true && > true && > (!false || false); I think we mostly care about 64b guest experience improvement here. So personally I am fine with your proposal. Also there is this vmalloc shortage issue, hit with aarch32 guest only, up to now (Which I reported at the end of the cover letter). This can cause some existing guest configs (even without FW) to not boot with the new high ECAM region whereas it booted before. I don't know if this is acceptable. Thanks Eric > > I see a return to the 0xCF8/0xCFC pattern regressive; I'd rather > restrict the large/high ECAM feature to 64-bit guests (with or without > firmware), and to 32-bit LPAE kernels that are launched without firmware > (which, I think, has been the case for most of their history). > > Personally I don't have a stake in 32-bit ARM, so do take my opinion > with a grain of salt. Wearing my upstream ArmVirtQemu co-maintainer hat, > my sole 32-bit interest is in keeping command lines working, *if* they > once worked. Not extending new QEMU features to 32-bit firmware is fine > with me -- in fact I would value that over seeing more quirky firmware > code just for 32-bit's sake. > > Side topic: the last subcondition basically says, "IF we use firmware > THEN the VM had better be 64-bit". This is a "logical implication": > A-->B. The C language doesn't have an "implication operator", so I > rewrote it equivalently with the logical negation and logical OR > operators: A-->B is equivalent to (!A || B). (If A is true, then B must > hold; if A is false, then B doesn't matter.) > > Thanks, > Laszlo > From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:49975) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1fLvsr-0008PO-Es for qemu-devel@nongnu.org; Thu, 24 May 2018 15:26:46 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1fLvsq-0008IK-8M for qemu-devel@nongnu.org; Thu, 24 May 2018 15:26:45 -0400 References: <1527091418-11874-1-git-send-email-eric.auger@redhat.com> <22c4e504-a7b4-e6dd-b2cc-618d306b6f0c@redhat.com> <550464ac-5155-6311-d0f7-92c7c0813f82@redhat.com> From: Auger Eric Message-ID: Date: Thu, 24 May 2018 21:26:32 +0200 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit Subject: Re: [Qemu-devel] [RFC 0/2] ARM virt: Support up to 256 PCIe buses List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Laszlo Ersek , Ard Biesheuvel Cc: Wei Huang , Peter Maydell , Andrew Jones , QEMU Developers , qemu-arm , Shannon Zhao , Eric Auger On 05/24/2018 07:20 PM, Laszlo Ersek wrote: > On 05/24/18 16:14, Ard Biesheuvel wrote: >> On 24 May 2018 at 15:59, Laszlo Ersek wrote: >>> On 05/24/18 15:07, Peter Maydell wrote: >>>> On 24 May 2018 at 13:59, Laszlo Ersek wrote: >>>>> On 05/24/18 11:11, Peter Maydell wrote: >>>>>> Won't it also break a guest which is just Linux loaded not via >>>>>> firmware which is an aarch32 kernel without LPAE support? >>>>> >>>>> Does such a thing exist? (I honestly have no clue.) >>>> >>>> Yes, it does; LPAE isn't a mandatory kernel config option. >>>> This is why we have the machine 'highmem' option, so that >>>> we can run on those kernels by not putting anything above >>>> the 4G boundary. Looking back at the history on that, we >>>> opted at the time for "default to highmem on, and if you're >>>> running an non-lpae kernel you need to turn it off manually". >>> >>> Ah, OK, I didn't know that. >>> >>>> So we can handle those kernels by just not putting ECAM >>>> above 4G if highmem is false. >>> >>> The problem is we can have a combination of 32-bit UEFI firmware (which >>> certainly lacks LPAE) and a 32-bit kernel which supports LPAE. >>> Previously, you wouldn't specify highmem=off, and things would just work >>> -- the firmware would simply ignore the >=4GB MMIO aperture, and use the >>> 32-bit MMIO aperture only (and use the sole 32-bit ECAM). The kernel >>> could then use both low and high MMIO apertures, however (I gather?). >>> >>> The difference with "high ECAM" is that it is *moved* (not *added*), so >>> the 32-bit firmware is left with nothing for config space access. For >>> booting the same combination as above, you are suddenly forced to add >>> highmem=off, just to keep the ECAM low -- and that, while it keeps the >>> firmware happy, prevents the LPAE-capable kernel from using the high >>> MMIO aperture. >>> >>> So I think "highmem_ecam" should be computed like this: >>> >>> highmem_ecam = highmem_ecam_machtype_default && >>> highmem && >>> (!firmware_loaded || aarch64); >>> >> >> Given that the firmware is tightly coupled to the platform, we may >> decide not to care about ECAM for UEFI itself, and invent a secondary >> config space access mechanism that does not consume such a huge amount >> of address space. For instance, legacy PCI uses a pair of I/O ports >> for this, one to set the address and one to perform the actual read or >> write, and we could easily implement something similar (such an >> interface is problematic in SMP context but we don't care about that >> anyway) >> >> Just a thought - perhaps we don't care enough about 32-bit to go >> through the trouble, but it would be nice if LPAE capable 32-bit >> guests could make use of the expanded PCIe config space as well. > > Under the above proposal, they could, they'd just have to be launched > without firmware: > > highmem_ecam_machtype_default = true; > highmem = true; > firmware_loaded = false; > aarch64 = false; > > highmem_ecam = true && > true && > (!false || false); I think we mostly care about 64b guest experience improvement here. So personally I am fine with your proposal. Also there is this vmalloc shortage issue, hit with aarch32 guest only, up to now (Which I reported at the end of the cover letter). This can cause some existing guest configs (even without FW) to not boot with the new high ECAM region whereas it booted before. I don't know if this is acceptable. Thanks Eric > > I see a return to the 0xCF8/0xCFC pattern regressive; I'd rather > restrict the large/high ECAM feature to 64-bit guests (with or without > firmware), and to 32-bit LPAE kernels that are launched without firmware > (which, I think, has been the case for most of their history). > > Personally I don't have a stake in 32-bit ARM, so do take my opinion > with a grain of salt. Wearing my upstream ArmVirtQemu co-maintainer hat, > my sole 32-bit interest is in keeping command lines working, *if* they > once worked. Not extending new QEMU features to 32-bit firmware is fine > with me -- in fact I would value that over seeing more quirky firmware > code just for 32-bit's sake. > > Side topic: the last subcondition basically says, "IF we use firmware > THEN the VM had better be 64-bit". This is a "logical implication": > A-->B. The C language doesn't have an "implication operator", so I > rewrote it equivalently with the logical negation and logical OR > operators: A-->B is equivalent to (!A || B). (If A is true, then B must > hold; if A is false, then B doesn't matter.) > > Thanks, > Laszlo >