From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 975DEC3DA4A for ; Mon, 29 Jul 2024 23:38:21 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:In-Reply-To:From:References:Cc:To:Subject:MIME-Version:Date: Message-ID:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=rYmAAB0qlkdPhNU6H9rrHKoah3EbC4V9mtuIUUVAOEw=; b=ncusbD4uEzg06hnG2R9LrWJvC8 vlPGbQQjIMopU23RjgCPButB/+HpkcE+sLfMweQIY+bCd31IN4rUZ24CcO9uOBaaJEJZZ8xBySGVg H51xSs6ZqAFt/157WUVn5Qy45D3Y0N6tlk4fgvaXNQN8/H40Bysq52FB9KAMWplsDI/tQBBDcZVrc GBxyg76LY5cA7egSEZqEakBxvGqHm5RCijWr8orUz+737ZK5z3YPssQ9hJ1rec+KrWX2zeDJxCWek e5wszmEDjFGsEgFdRcWcjM6GWiI6P3eSM1Boqex9nZ7axhDeASaXvaRlIIuPk6K7uGMKj6jb14Ed4 jbHAklMw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.97.1 #2 (Red Hat Linux)) id 1sYZwJ-0000000D1qz-3wkz; Mon, 29 Jul 2024 23:38:03 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.97.1 #2 (Red Hat Linux)) id 1sYZvr-0000000D1kv-2G7Q for linux-arm-kernel@lists.infradead.org; Mon, 29 Jul 2024 23:37:37 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1722296253; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=rYmAAB0qlkdPhNU6H9rrHKoah3EbC4V9mtuIUUVAOEw=; b=a7NBVWMSRsKrzU5ES69xGOcbMmjGZ6Xf4KJqiF4pwfQlexsTG5pSpLEDrDj9/fC8sa+vGV kMJx0DY1gtyYNz4yPs5BXn3K51NAh8gsJEN4nKuia8eSXzacwlmeV63VN1O2D+qpNX4Q4B agOPRcPevDgc1+vUT6+kIS6NDtv5/Xo= Received: from mail-oi1-f200.google.com (mail-oi1-f200.google.com [209.85.167.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-580-KWieQNHcN_6Zd-UMxnW1wQ-1; Mon, 29 Jul 2024 19:37:31 -0400 X-MC-Unique: KWieQNHcN_6Zd-UMxnW1wQ-1 Received: by mail-oi1-f200.google.com with SMTP id 5614622812f47-3db2b8a3ed6so2621085b6e.2 for ; Mon, 29 Jul 2024 16:37:31 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722296251; x=1722901051; h=content-transfer-encoding:in-reply-to:from:content-language :references:cc:to:subject:user-agent:mime-version:date:message-id :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=rYmAAB0qlkdPhNU6H9rrHKoah3EbC4V9mtuIUUVAOEw=; b=evOPHmgu3oXc4D8fmhzkSL4Y+xUakNW0zoEaPSgP3Q/czoEGOqYafY8lkS62QTRcLe 0Z6Nl06VNxxg0QzLitLDqZ0UxT+Dra5mqQWAytOmIfWzW6bTLfuwqpjQMEeTUae/3aTx iHdb62seXorA+Qb35OYgxPJ3T+2Sa3p+NZJ+Rnuj7TGr1X+YYbOfkf8BUEvkGa8cAinH i8TsSpN0E/Zxh9Mvu9TcB05sdjd5KDsrYYLYcz7g4OTeIpCT3bzj5u6ShW/BpI08rwRW YcrhWYdfgjwtNjjyNKUQHD9EzsLHVS4lH1hjZa5VxKgtaJOVUAw2Kg4t8GLQwqcWJRGe FLrQ== X-Forwarded-Encrypted: i=1; AJvYcCXRG14rpgimqLqgvQcQ3rrz7tMjsGu2Oano6pC+JROAwsaVJRxyaHsTZhSpI11iqnD4OaBbJoh2oDGV1Pyk9FVj6SQBvkqFdNX/23BEqgsRxyr9q58= X-Gm-Message-State: AOJu0YwDO+GSwLFSInOGyPqOOCP+CVOU3ORCiwpsDJ19L8JrIJdgWT1n T+TNyiOEcF2I+j+29HWCRnBiaLn9apvQrRbqjW4X5yMOA9t/vlrZPrIRizcXAMfGxWTuLblyR7G IT8Q42M/c3bVvl455Tu4MStXWc72AfE5LTVAb7kZLAWU+2b75vjFiKYkj3REY/PcsbdzmZE8P X-Received: by 2002:a05:6808:bd5:b0:3d9:22a4:d5f3 with SMTP id 5614622812f47-3db239d09a4mr13603402b6e.21.1722296250652; Mon, 29 Jul 2024 16:37:30 -0700 (PDT) X-Google-Smtp-Source: AGHT+IETOHCPOcvWc5jhOBqkV2140PExLjmylzs4Ybfkt9mudPgH3Q9MtP9S9MSCOGtMrFMDNMPzbg== X-Received: by 2002:a05:6808:bd5:b0:3d9:22a4:d5f3 with SMTP id 5614622812f47-3db239d09a4mr13603373b6e.21.1722296250208; Mon, 29 Jul 2024 16:37:30 -0700 (PDT) Received: from [192.168.68.54] ([43.252.112.134]) by smtp.gmail.com with ESMTPSA id 41be03b00d2f7-7a9fa5a3760sm7618434a12.94.2024.07.29.16.37.22 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 29 Jul 2024 16:37:29 -0700 (PDT) Message-ID: <2b4f0496-99f4-4bc6-af6c-a8be8fca69a8@redhat.com> Date: Tue, 30 Jul 2024 09:37:19 +1000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v4 03/15] arm64: Detect if in a realm and set RIPAS RAM To: Steven Price , kvm@vger.kernel.org, kvmarm@lists.linux.dev Cc: Suzuki K Poulose , Catalin Marinas , Marc Zyngier , Will Deacon , James Morse , Oliver Upton , Zenghui Yu , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, Joey Gouly , Alexandru Elisei , Christoffer Dall , Fuad Tabba , linux-coco@lists.linux.dev, Ganapatrao Kulkarni References: <20240701095505.165383-1-steven.price@arm.com> <20240701095505.165383-4-steven.price@arm.com> From: Gavin Shan In-Reply-To: <20240701095505.165383-4-steven.price@arm.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20240729_163735_771388_EB8DDF6D X-CRM114-Status: GOOD ( 42.42 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 7/1/24 7:54 PM, Steven Price wrote: > From: Suzuki K Poulose > > Detect that the VM is a realm guest by the presence of the RSI > interface. > > If in a realm then all memory needs to be marked as RIPAS RAM initially, > the loader may or may not have done this for us. To be sure iterate over > all RAM and mark it as such. Any failure is fatal as that implies the > RAM regions passed to Linux are incorrect - which would mean failing > later when attempting to access non-existent RAM. > > Signed-off-by: Suzuki K Poulose > Co-developed-by: Steven Price > Signed-off-by: Steven Price > --- > Changes since v3: > * Provide safe/unsafe versions for converting memory to protected, > using the safer version only for the early boot. > * Use the new psci_early_test_conduit() function to avoid calling an > SMC if EL3 is not present (or not configured to handle an SMC). > Changes since v2: > * Use DECLARE_STATIC_KEY_FALSE rather than "extern struct > static_key_false". > * Rename set_memory_range() to rsi_set_memory_range(). > * Downgrade some BUG()s to WARN()s and handle the condition by > propagating up the stack. Comment the remaining case that ends in a > BUG() to explain why. > * Rely on the return from rsi_request_version() rather than checking > the version the RMM claims to support. > * Rename the generic sounding arm64_setup_memory() to > arm64_rsi_setup_memory() and move the call site to setup_arch(). > --- > arch/arm64/include/asm/rsi.h | 64 +++++++++++++++++++++++++ > arch/arm64/include/asm/rsi_cmds.h | 22 +++++++++ > arch/arm64/kernel/Makefile | 3 +- > arch/arm64/kernel/rsi.c | 77 +++++++++++++++++++++++++++++++ > arch/arm64/kernel/setup.c | 8 ++++ > 5 files changed, 173 insertions(+), 1 deletion(-) > create mode 100644 arch/arm64/include/asm/rsi.h > create mode 100644 arch/arm64/kernel/rsi.c > > diff --git a/arch/arm64/include/asm/rsi.h b/arch/arm64/include/asm/rsi.h > new file mode 100644 > index 000000000000..29fdc194d27b > --- /dev/null > +++ b/arch/arm64/include/asm/rsi.h > @@ -0,0 +1,64 @@ > +/* SPDX-License-Identifier: GPL-2.0-only */ > +/* > + * Copyright (C) 2024 ARM Ltd. > + */ > + > +#ifndef __ASM_RSI_H_ > +#define __ASM_RSI_H_ > + > +#include > +#include > + > +DECLARE_STATIC_KEY_FALSE(rsi_present); > + > +void __init arm64_rsi_init(void); > +void __init arm64_rsi_setup_memory(void); > +static inline bool is_realm_world(void) > +{ > + return static_branch_unlikely(&rsi_present); > +} > + > +static inline int rsi_set_memory_range(phys_addr_t start, phys_addr_t end, > + enum ripas state, unsigned long flags) > +{ > + unsigned long ret; > + phys_addr_t top; > + > + while (start != end) { > + ret = rsi_set_addr_range_state(start, end, state, flags, &top); > + if (WARN_ON(ret || top < start || top > end)) > + return -EINVAL; > + start = top; > + } > + > + return 0; > +} > + @flags has been defined as int instead of unsigned long, which is inconsistent to TF-RMM's definitions since it has type of 'unsigned long'. > +/* > + * Convert the specified range to RAM. Do not use this if you rely on the > + * contents of a page that may already be in RAM state. > + */ > +static inline int rsi_set_memory_range_protected(phys_addr_t start, > + phys_addr_t end) > +{ > + return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, > + RSI_CHANGE_DESTROYED); > +} > + > +/* > + * Convert the specified range to RAM. Do not convert any pages that may have > + * been DESTROYED, without our permission. > + */ > +static inline int rsi_set_memory_range_protected_safe(phys_addr_t start, > + phys_addr_t end) > +{ > + return rsi_set_memory_range(start, end, RSI_RIPAS_RAM, > + RSI_NO_CHANGE_DESTROYED); > +} > + > +static inline int rsi_set_memory_range_shared(phys_addr_t start, > + phys_addr_t end) > +{ > + return rsi_set_memory_range(start, end, RSI_RIPAS_EMPTY, 0); > +} > +#endif s/0/RSI_NO_CHANGE_DESTROYED s/#endif/#endif /* __ASM_RSI_H_ */ > diff --git a/arch/arm64/include/asm/rsi_cmds.h b/arch/arm64/include/asm/rsi_cmds.h > index 89e907f3af0c..acb557dd4b88 100644 > --- a/arch/arm64/include/asm/rsi_cmds.h > +++ b/arch/arm64/include/asm/rsi_cmds.h > @@ -10,6 +10,11 @@ > > #include > > +enum ripas { > + RSI_RIPAS_EMPTY, > + RSI_RIPAS_RAM, > +}; > + > static inline unsigned long rsi_request_version(unsigned long req, > unsigned long *out_lower, > unsigned long *out_higher) > @@ -35,4 +40,21 @@ static inline unsigned long rsi_get_realm_config(struct realm_config *cfg) > return res.a0; > } > > +static inline unsigned long rsi_set_addr_range_state(phys_addr_t start, > + phys_addr_t end, > + enum ripas state, > + unsigned long flags, > + phys_addr_t *top) > +{ > + struct arm_smccc_res res; > + > + arm_smccc_smc(SMC_RSI_IPA_STATE_SET, start, end, state, > + flags, 0, 0, 0, &res); > + > + if (top) > + *top = res.a1; > + > + return res.a0; > +} > + > #endif > diff --git a/arch/arm64/kernel/Makefile b/arch/arm64/kernel/Makefile > index 763824963ed1..a483b916ed11 100644 > --- a/arch/arm64/kernel/Makefile > +++ b/arch/arm64/kernel/Makefile > @@ -33,7 +33,8 @@ obj-y := debug-monitors.o entry.o irq.o fpsimd.o \ > return_address.o cpuinfo.o cpu_errata.o \ > cpufeature.o alternative.o cacheinfo.o \ > smp.o smp_spin_table.o topology.o smccc-call.o \ > - syscall.o proton-pack.o idle.o patching.o pi/ > + syscall.o proton-pack.o idle.o patching.o pi/ \ > + rsi.o > > obj-$(CONFIG_COMPAT) += sys32.o signal32.o \ > sys_compat.o > diff --git a/arch/arm64/kernel/rsi.c b/arch/arm64/kernel/rsi.c > new file mode 100644 > index 000000000000..f01bff9dab04 > --- /dev/null > +++ b/arch/arm64/kernel/rsi.c > @@ -0,0 +1,77 @@ > +// SPDX-License-Identifier: GPL-2.0-only > +/* > + * Copyright (C) 2023 ARM Ltd. > + */ > + > +#include > +#include > +#include > +#include > + > +DEFINE_STATIC_KEY_FALSE_RO(rsi_present); > +EXPORT_SYMBOL(rsi_present); > + > +static bool rsi_version_matches(void) > +{ > + unsigned long ver_lower, ver_higher; > + unsigned long ret = rsi_request_version(RSI_ABI_VERSION, > + &ver_lower, > + &ver_higher); > + > + if (ret == SMCCC_RET_NOT_SUPPORTED) > + return false; > + > + if (ret != RSI_SUCCESS) { > + pr_err("RME: RMM doesn't support RSI version %u.%u. Supported range: %lu.%lu-%lu.%lu\n", > + RSI_ABI_VERSION_MAJOR, RSI_ABI_VERSION_MINOR, > + RSI_ABI_VERSION_GET_MAJOR(ver_lower), > + RSI_ABI_VERSION_GET_MINOR(ver_lower), > + RSI_ABI_VERSION_GET_MAJOR(ver_higher), > + RSI_ABI_VERSION_GET_MINOR(ver_higher)); > + return false; > + } > + > + pr_info("RME: Using RSI version %lu.%lu\n", > + RSI_ABI_VERSION_GET_MAJOR(ver_lower), > + RSI_ABI_VERSION_GET_MINOR(ver_lower)); > + > + return true; > +} > + > +void __init arm64_rsi_setup_memory(void) > +{ > + u64 i; > + phys_addr_t start, end; > + > + if (!is_realm_world()) > + return; > + > + /* > + * Iterate over the available memory ranges and convert the state to ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ blocks and convert them to > + * protected memory. We should take extra care to ensure that we DO NOT > + * permit any "DESTROYED" pages to be converted to "RAM". > + * > + * BUG_ON is used because if the attempt to switch the memory to > + * protected has failed here, then future accesses to the memory are > + * simply going to be reflected as a fault which we can't handle. > + * Bailing out early prevents the guest limping on and dieing later. > + */ > + for_each_mem_range(i, &start, &end) { > + BUG_ON(rsi_set_memory_range_protected_safe(start, end)); > + } > +} > + If I'm understanding the code completely, this changes the memory state from RIPAS_EMPTY to RIPAS_RAM so that the following page faults can be routed to host properly. Otherwise, a SEA is injected to the realm according to tf-rmm/runtime/core/exit.c::handle_data_abort(). The comments can be more explicit to replace "fault" with "SEA (Synchronous External Abort)". Besides, this forces a guest exit with reason RMI_EXIT_RIPAS_CHANGE which is handled by the host, where RMI_RTT_SET_RIPAS is triggered to convert the memory state from RIPAS_EMPTY to RIPAS_RAM. The question is why the conversion can't be done by VMM (QEMU)? > +void __init arm64_rsi_init(void) > +{ > + /* > + * If PSCI isn't using SMC, RMM isn't present. Don't try to execute an > + * SMC as it could be UNDEFINED. > + */ > + if (!psci_early_test_conduit(SMCCC_CONDUIT_SMC)) > + return; > + if (!rsi_version_matches()) > + return; > + > + static_branch_enable(&rsi_present); > +} > + > diff --git a/arch/arm64/kernel/setup.c b/arch/arm64/kernel/setup.c > index a096e2451044..143f87615af0 100644 > --- a/arch/arm64/kernel/setup.c > +++ b/arch/arm64/kernel/setup.c > @@ -43,6 +43,7 @@ > #include > #include > #include > +#include > #include > #include > #include > @@ -293,6 +294,11 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) > * cpufeature code and early parameters. > */ > jump_label_init(); > + /* > + * Init RSI before early param so that "earlycon" console uses the > + * shared alias when in a realm > + */ > + arm64_rsi_init(); > parse_early_param(); > > dynamic_scs_init(); > @@ -328,6 +334,8 @@ void __init __no_sanitize_address setup_arch(char **cmdline_p) > > arm64_memblock_init(); > > + arm64_rsi_setup_memory(); > + > paging_init(); > > acpi_table_upgrade(); Thanks, Gavin