From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.6 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_2 autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 809F9C433B4 for ; Mon, 17 May 2021 12:22:11 +0000 (UTC) Received: from desiato.infradead.org (desiato.infradead.org [90.155.92.199]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id D37C961242 for ; Mon, 17 May 2021 12:22:10 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D37C961242 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=Huawei.com Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=desiato.20200630; h=Sender:Content-Transfer-Encoding :Content-Type:List-Subscribe:List-Help:List-Post:List-Archive: List-Unsubscribe:List-Id:MIME-Version:References:In-Reply-To:Message-ID: Subject:CC:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=yy9p3DmwQUJ8JNdUDCjsU5x7MISz/JjrBBUy5zD4WW4=; b=C3TFfOhEI9dXkDe4x4JFCEp22 eali4NWZWi8a8W/7e1QRocGbPR2+mszZFHoH6OxAjuZLPWzQrNA5dDG0xSrpYQOZ2mNVRDJLkhnVY d63dkcz+YfJ6gFzwqUY8yYLKXdwdUd0hOz0CDlHVLDpReZuIpc1qVS6c2NYFXePw/Xr6wdWLq6RGx 8ONhbNbBGeOvvYrAenwZ30RSUTGMJtAjQxU2do+Qhb/rU97YNB+aFLrl72r1i3odMSblY72fdI8mb kfdHx5TV+AO/encH3KBSCmbgO7rYtY/86EtgdhNFlp9CvxsyLrr2a4Pu91uLQN6GO72Gf5dAdIc9R SDtLZXaig==; Received: from localhost ([::1] helo=desiato.infradead.org) by desiato.infradead.org with esmtp (Exim 4.94 #2 (Red Hat Linux)) id 1licDh-00Es2p-Lg; Mon, 17 May 2021 12:19:38 +0000 Received: from bombadil.infradead.org ([2607:7c80:54:e::133]) by desiato.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1licDU-00Es1N-9Z for linux-arm-kernel@desiato.infradead.org; Mon, 17 May 2021 12:19:31 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20210309; h=Content-Transfer-Encoding: Content-Type:MIME-Version:References:In-Reply-To:Message-ID:Subject:CC:To: From:Date:Sender:Reply-To:Content-ID:Content-Description; bh=2AuhZG9uIerVJAs1xoKrkBAlBiRygrkeg6cmZX/UIqs=; b=cRZzGTMgJtZxpy+YRJQxVeb+hY 02rIhrTpT0BbdCyj3TeY3bI//ZPiuF7YWB9zye+vgP62zY87mSakppfQb+tDaAwLLAyhRBkHDzhSL SpJAbc+hDjbotf/6ZG+FrLyCb2FR+zLVxDFnbsksyaybdXHj6Gf/rGXZOYEuP1K1IqgQ27SM9JtIM ElKvbOE0QHr6eg15EbBOW8kr/hpDb8LMWdEiitjFh1WWAl4w+GcS4IZVz6QtCgycDWhbsaxZg1jR8 O/apnA16KKYpNDZyw/+5TpPGBVmPAoj77KnhI6Zec/La/E7Hh0AkUxx/2g5NWRKXLWnyWSwWz3Chg c/3PV1cw==; Received: from szxga04-in.huawei.com ([45.249.212.190]) by bombadil.infradead.org with esmtps (Exim 4.94 #2 (Red Hat Linux)) id 1licDO-00DkX0-N1 for linux-arm-kernel@lists.infradead.org; Mon, 17 May 2021 12:19:23 +0000 Received: from dggems704-chm.china.huawei.com (unknown [172.30.72.60]) by szxga04-in.huawei.com (SkyGuard) with ESMTP id 4FkJ4f2hk5zmhRG; Mon, 17 May 2021 20:15:46 +0800 (CST) Received: from lhreml710-chm.china.huawei.com (10.201.108.61) by dggems704-chm.china.huawei.com (10.3.19.181) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 17 May 2021 20:19:12 +0800 Received: from localhost (10.52.123.135) by lhreml710-chm.china.huawei.com (10.201.108.61) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2176.2; Mon, 17 May 2021 13:19:10 +0100 Date: Mon, 17 May 2021 13:17:25 +0100 From: Jonathan Cameron To: Ard Biesheuvel CC: James Morse , Linux ARM , Catalin Marinas , Will Deacon , Subject: Re: [RFC PATCH] Documentation/arm64: describe the kernel's expectations of 'memory' Message-ID: <20210517131725.00002068@Huawei.com> In-Reply-To: References: <20210517103319.5356-1-james.morse@arm.com> <20210517122701.00005e19@Huawei.com> Organization: Huawei Technologies Research and Development (UK) Ltd. X-Mailer: Claws Mail 3.17.8 (GTK+ 2.24.33; i686-w64-mingw32) MIME-Version: 1.0 X-Originating-IP: [10.52.123.135] X-ClientProxiedBy: lhreml739-chm.china.huawei.com (10.201.108.189) To lhreml710-chm.china.huawei.com (10.201.108.61) X-CFilter-Loop: Reflected X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210517_051919_116898_16CE73F5 X-CRM114-Status: GOOD ( 53.22 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, 17 May 2021 13:55:16 +0200 Ard Biesheuvel wrote: > On Mon, 17 May 2021 at 13:30, Jonathan Cameron > wrote: > > > > On Mon, 17 May 2021 11:33:19 +0100 > > James Morse wrote: > > > > > Standards such as CXL allow memory on PCIe devices to be made > > > available to the operating system for use as regular memory. > > > > > > Document linux's expectations around the behaviour of memory as the > > > implementations of these new standards may need special treatment in > > > the OS, firmware or bootloader. > > > > > > Signed-off-by: James Morse > > > > Hi James, > > > > +CC linux-cxl to pick up a few more interesting people who might loose > > this in the wash of linux-arm-kernel > > > > Good to see this description as there has been some confusion on this > > point. This basically looks like what I'd expect to see. Just a few > > comments around firmware description towards the end. > > > > > --- > > > Documentation/arm64/memory.rst | 31 +++++++++++++++++++++++++++++++ > > > 1 file changed, 31 insertions(+) > > > > > > diff --git a/Documentation/arm64/memory.rst b/Documentation/arm64/memory.rst > > > index 901cd094f4ec..951802aee55f 100644 > > > --- a/Documentation/arm64/memory.rst > > > +++ b/Documentation/arm64/memory.rst > > > @@ -167,3 +167,34 @@ from a 52-bit space by enabling the following kernel config options: > > > > > > Note that this option is only intended for debugging applications > > > and should not be used in production. > > > + > > > +On device memory used as regular memory > > > +--------------------------------------- > > > +Standards such as CXL allow memory on PCIe device to be made > > > +available to the operating system for use as regular memory. > > > + > > > +If memory is added to the UEFI memory map or DT, or discovered via ACPI's SRAT, > > > +linux expects it to function in the same way as the bulk DRAM. This section > > > > Linux > > > > > +terms this 'regular memory'. > > > + > > > +The kernel may use any attributes to map this memory, e.g. Device-nGnRnE or > > > +Normal Writeback-Cacheable. The kernel may not be in control of the attributes > > > +used, e.g. if the memory is used by a KVM guest. > > > +The kernel will perform cache maintenance to resolve mismatched attributes, > > > +e.g. invalidating clean stale lines after writing new data when the MMU is > > > +disabled. > > > + > > > +The memory may be used by any instruction supported by the CPUs. > > > +e.g. Even when the v8.1 LSE atomic instructions are supported, the v8.0 > > > +exclusives are still used for the futex code, and conditional waits, and still > > > +used by existing user-space binaries. When the CPUs support features such as > > > +MTE, all regular memory must support MTE tags. > > > + > > > +On device memory that does not function in the same way as regular memory must > > > +not be added to the UEFI memory map or DT, or be discovered via ACPI's SRAT. > > > + > > > +On arm64, the kernel does not rewrite the UEFI memory map when memory is added > > > +or removed. On device memory that is present at boot, but must be removed later > > > > Might be worth giving an example of why memory 'must be removed'? I'm not sure > > what you are getting at there. Specific purpose memory? > > > > > +should be discovered via ACPI's SRAT to ensure it is not used for non-movable > > > +structures. > > > > Not sure I follow this part. It could be of type EFI_MEMORY_SP. > > EFI_MEMORY_SP is an attribute, not a type. Good point. > > > It should be in SRAT as well, but the EFI type should be sufficient to avoid > > problems. > > "The SPM attribute serves as a hint to the OS to avoid allocating this memory > > for core OS data or code that can not be relocated." > > > > Now I'm not sure the kernel is handling EFI_MEMORY_SP fully yet... If > > we need to exclude this approach for now, then this text should perhaps > > call it out explicitly. > > > > The problem with EFI_MEMORY_SP is that it is not a type, but an > attribute, which gives a hint to the OS about the nature of the > memory, which the OS is free to ignore. IIRC the way around that is to use the reserved type + EFI_MEMORY_SP. An unware bootloader or OS will then not use it and hence we are safe. An aware driver can then decide it is safe to "hotplug" said memory. > > The UEFI memory map is not only consumed by the OS, but by any driver > or OS loader that executes in the EFI boot environment, e.g., GPU > drivers or shim/grub bootloaders. If these are not enlightened and > understand what EFI_MEMORY_SP means, they may (and are entitled to) > treat this EFI_MEMORY_SP as if it were regular memory. If GRUB loads > the kernel into EFI_MEMORY_SP memory, it had better behave like > regular memory or things will fall apart. Two separate issues here. The 'broken' one where _SP or indeed hotplug flag is no use, and the one where it is 'must be removed later' and we just don't want to put unmovable allocations in it. > > This means that EFI_MEMORY_SP is really only suitable to describe > aspects of the memory range that can be happily ignored. MTE or > atomics capability must be described in a different way. > That's indeed the intent. These are just hints and indeed not suitable for the cases where things are broken (MTE / Atomics). In those you should not be claiming it is normal memory at all. SRAT doesn't help you with that though. The hotplug flag is SRAT is also only a hint. OS doesn't have to take any notice or support it nor does any boot loader. Things will 'work' with the exception of hot-remove. If you definitely don't want your memory to be used by the OS for normal purposes, then don't present it in a form where it might be. Jonathan > > > > > +e.g. the kernel text, page tables or the GIC ITS Pending Table. > > > > > > _______________________________________________ > > linux-arm-kernel mailing list > > linux-arm-kernel@lists.infradead.org > > http://lists.infradead.org/mailman/listinfo/linux-arm-kernel _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel