From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 82F03EB64D9 for ; Thu, 29 Jun 2023 11:31:54 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B83458D0003; Thu, 29 Jun 2023 07:31:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B33FB8D0001; Thu, 29 Jun 2023 07:31:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9D4C28D0003; Thu, 29 Jun 2023 07:31:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 8C6118D0001 for ; Thu, 29 Jun 2023 07:31:53 -0400 (EDT) Received: from smtpin21.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 398FD80EBF for ; Thu, 29 Jun 2023 11:31:53 +0000 (UTC) X-FDA: 80955570906.21.8875421 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf24.hostedemail.com (Postfix) with ESMTP id AE84918001C for ; Thu, 29 Jun 2023 11:31:50 +0000 (UTC) Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DwezoMkq; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688038310; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GoRf97kbYDZBwP3J/hRLkmLH66DPLZ1hXk0cIFzlfrw=; b=3lHvp6ns1FH0AjckQtY6Xif1wq45mrHP+osbkKs5foLoUxXh8lAtyKHcZMs4gxXkN/lPOb wMSvy0SzfKgXVHOqF+r7nSDq133cHs8y+g42xcdRQ6QloSqqd2qvabjThGIjqXF+ztDBZ1 Gd0fP1x0wu43rqOMwlF3SRa4+xgoKGw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688038310; a=rsa-sha256; cv=none; b=RSfDSJ7liRMBBhmDCRrvAL+N811vj/UFAk0AXBAEJ75KocNWN9JFjrk48xWLiiYHmqTKdh KUDzSY74E+nuAf4quRmvaSYCqoqvxRa2rw4YwitnUOeroU7qDb9CpczbIEu7hjY3ETqY1W 4mcHK/X8u/4U/W6UtydLi1ev1B7nNL0= ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=DwezoMkq; spf=pass (imf24.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688038309; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=GoRf97kbYDZBwP3J/hRLkmLH66DPLZ1hXk0cIFzlfrw=; b=DwezoMkq9JVcL2Mro7HWODGWpf+jX7uOkpek6RMMc4GfWq84DhOeUE3FAauqIGm4SeCiw4 Pz5EPYmzhBpzr33vSRlqpMfySl1JgGWeFzo9ES38a/Wu6Grmb7++NcuosY6PbGCMEey23u tPgnCu7WIwTQ5H68zYy4sniEyGD2U60= Received: from mail-lf1-f69.google.com (mail-lf1-f69.google.com [209.85.167.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-609-3spHcfTvN9Sr392q1Df-1g-1; Thu, 29 Jun 2023 07:31:48 -0400 X-MC-Unique: 3spHcfTvN9Sr392q1Df-1g-1 Received: by mail-lf1-f69.google.com with SMTP id 2adb3069b0e04-4f76712f950so581484e87.0 for ; Thu, 29 Jun 2023 04:31:47 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688038306; x=1690630306; h=content-transfer-encoding:in-reply-to:organization:from:references :cc:to:content-language:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=GoRf97kbYDZBwP3J/hRLkmLH66DPLZ1hXk0cIFzlfrw=; b=iYXvfGikySoPhboBj7xUwtFazqeXYy3/7I3LQxCHSfanQOT45HvKDT1KTXYwdPW0+v 47vbwckwR9tPzK1OIetpTjA07nSmGzki9HQNxXdNB13pikCrhKeYiQ+2k61Khj7fBVug DH3wYTqz/P3HG0XD57tpsGjrVCcDXteIgMz853lV9Hc+W3dsGiNftZ7Br74V99OgUC8u HSzBaMUs36LnBRaYSvSdBii6G9Z84NUbDj6cZhUWIzRK8Qi/oczgLsrZOdWfONz3kI5F Ptg4yycOvbxwcgdVpHHauTwYub1ApPBwye48P7sc2Yc20R2IrkEEwoCLZGfwaq/X8BFb iNHg== X-Gm-Message-State: ABy/qLbA4mNhCVVF26MCH7zGBJlVbDnjM0GFxpusn8U7w6Ed6LaW4C2M 4NeqeUT7k6nLgNNKeaJECMMKeONCN1OI3aMjLX4YQ1+/ngM9ehmAv7FO36zkeTsoi46LBACpki4 QpNOXlUjdON0= X-Received: by 2002:ac2:4c48:0:b0:4fb:9595:15e9 with SMTP id o8-20020ac24c48000000b004fb959515e9mr3245801lfk.22.1688038305954; Thu, 29 Jun 2023 04:31:45 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ759QKQQgqTCxa6IKWpXNlinwcvLHmiY40QmBJrFlDbDPxQTAATynaKDRxhNlc/1jVC7j3A+Q== X-Received: by 2002:ac2:4c48:0:b0:4fb:9595:15e9 with SMTP id o8-20020ac24c48000000b004fb959515e9mr3245756lfk.22.1688038305499; Thu, 29 Jun 2023 04:31:45 -0700 (PDT) Received: from ?IPV6:2a09:80c0:192:0:5dac:bf3d:c41:c3e7? ([2a09:80c0:192:0:5dac:bf3d:c41:c3e7]) by smtp.gmail.com with ESMTPSA id hn8-20020a05600ca38800b003fa722e8b48sm19303141wmb.32.2023.06.29.04.31.44 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 29 Jun 2023 04:31:45 -0700 (PDT) Message-ID: Date: Thu, 29 Jun 2023 13:31:43 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.12.0 Subject: Re: [PATCH v12 07/22] x86/virt/tdx: Add skeleton to enable TDX on demand To: Kai Huang , linux-kernel@vger.kernel.org, kvm@vger.kernel.org Cc: linux-mm@kvack.org, x86@kernel.org, dave.hansen@intel.com, kirill.shutemov@linux.intel.com, tony.luck@intel.com, peterz@infradead.org, tglx@linutronix.de, bp@alien8.de, mingo@redhat.com, hpa@zytor.com, seanjc@google.com, pbonzini@redhat.com, dan.j.williams@intel.com, rafael.j.wysocki@intel.com, ashok.raj@intel.com, reinette.chatre@intel.com, len.brown@intel.com, ak@linux.intel.com, isaku.yamahata@intel.com, ying.huang@intel.com, chao.gao@intel.com, sathyanarayanan.kuppuswamy@linux.intel.com, nik.borisov@suse.com, bagasdotme@gmail.com, sagis@google.com, imammedo@redhat.com References: <104d324cd68b12e14722ee5d85a660cccccd8892.1687784645.git.kai.huang@intel.com> From: David Hildenbrand Organization: Red Hat In-Reply-To: <104d324cd68b12e14722ee5d85a660cccccd8892.1687784645.git.kai.huang@intel.com> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: AE84918001C X-Rspam-User: X-Stat-Signature: pbc9jstpcat4bonwqhgoi9odauunq4yi X-Rspamd-Server: rspam03 X-HE-Tag: 1688038310-618751 X-HE-Meta: U2FsdGVkX1+DZDkrP2Byip3JvFCW47PlWwMUNbVv1uWrxJ4W+FhkVIUgioMiKq2iw+AygtwROSBYlHQaTjg0eYXzPQCHwaU7DDjJoJXjLOkL86jTfMbc9QRnRTYHEwM9JAbjUFct6MQCMTe2sD42dCfnkGOa3xs5irNSQWD161I/Zs+sFt1JFpPLbrypO48dGEdA8D81GLhSXIpfVQ4xJhqU5/aCeHQiY3ZzrsWN2pUwZQ3Ypg9n7A81Yrn0i6TNomFItQUd2jEkp1Abc4WV+pdLR0IrhCBUVWRtCrIz82t5lKuWdgR3zpiAgcfW7A1MoZUGcSiAyNnfo7l0rSdQ+ipnsjl/6iYbnQtMstlc7RaIZhVcUSJsNvrsqGIqWiqNoNajh6ECLnBbVlAXiAG0q17YwYa/AF5P4QY6AugTv2CZKyLP/5JKTPfh7Q+S1mve8UEUo0rXPbodJjY9HD8BVYveOHz4fwKLPfxkVRE9p6cZ8o9SWKUsjlAAkS5qc4ho1CU5KsZYZaTlVwCpdIIktGPDNxdQu3eF6FOkHp7bE6cpj4cS8FYIxpT7rGEM9T+WilbF2ojT1AQwAkNcr0TUNTcg6HJFSkdfG73KGtW9VeR1bQ9BfyXaSQid6xXfY9fcu1PFtxLzkQLuP+T9UzOPYbWRGn3rL4ohwz4dHyy2ZpKsryT050ELVxRW9V0RsUDFVaRzHG5xERM9s1iTI1QMukn+y8lGKIklGW0pyRJLQijIuAMTCTxB+KGoh8fvqo69ruXjImagDt6IKAw3yC9pUWNG/AM2H0qUzlys4twtbhaLvhZSqaUW6HwUs3PgLdjpkJtYE3BZD83xeJ4oaMYwx1lvXWz/RCHdGUdJH4+/S1ummqoat/1l0Awvz1Snyw/wGjI5zzIce5z7VNTZS0lE9X8/YDeScLEGw9kvCT5II0SwfBghExsiUMwrGokV5+fDuRDfJ3QacWQhtJppHYO IQSPofn2 EFLXGXvDonnVhflxZfNjzyYyegWxS7FI91JcnwYAo5Hh+9lA0fijrnUBia4AfIvxBfHBBg71xNeWV0hKPZiCdNLpxNco4E0BCDsAnBkk9ZLlbGlooHu1dBdJkTG1Y+HEUFaIwRXizDfXF5i1CrWgGgHw63TpsirDG9ud3xeTDhdPEscAFX8fT1v/4/BkaVmFgNO/S893N+1fdQ/fiT4Lzv1Ahp3n6O9ZhCSNzfZ5geaDXdz/rXcrcGX/R4JvFebI7I63ArxIYXGgQtboY3MmFx0sNHQVDTe7N426Ly6Z/BKYZAfGEBCBSeP2t8fFrJZ6jWQemEb8VYgXfUpzGPdjFjpzm8LyDkNpdyv3M7mPqirJqsDfsY3uGyE6zkbfb+nqR6T0nihifbSHp8V+foA4X3u6VoWj41tjXk2D/BuL9Vpy65fcsRPJ+VWxvgg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 26.06.23 16:12, Kai Huang wrote: > To enable TDX the kernel needs to initialize TDX from two perspectives: > 1) Do a set of SEAMCALLs to initialize the TDX module to make it ready > to create and run TDX guests; 2) Do the per-cpu initialization SEAMCALL > on one logical cpu before the kernel wants to make any other SEAMCALLs > on that cpu (including those involved during module initialization and > running TDX guests). > > The TDX module can be initialized only once in its lifetime. Instead > of always initializing it at boot time, this implementation chooses an > "on demand" approach to initialize TDX until there is a real need (e.g > when requested by KVM). This approach has below pros: > > 1) It avoids consuming the memory that must be allocated by kernel and > given to the TDX module as metadata (~1/256th of the TDX-usable memory), > and also saves the CPU cycles of initializing the TDX module (and the > metadata) when TDX is not used at all. > > 2) The TDX module design allows it to be updated while the system is > running. The update procedure shares quite a few steps with this "on > demand" initialization mechanism. The hope is that much of "on demand" > mechanism can be shared with a future "update" mechanism. A boot-time > TDX module implementation would not be able to share much code with the > update mechanism. > > 3) Making SEAMCALL requires VMX to be enabled. Currently, only the KVM > code mucks with VMX enabling. If the TDX module were to be initialized > separately from KVM (like at boot), the boot code would need to be > taught how to muck with VMX enabling and KVM would need to be taught how > to cope with that. Making KVM itself responsible for TDX initialization > lets the rest of the kernel stay blissfully unaware of VMX. > > Similar to module initialization, also make the per-cpu initialization > "on demand" as it also depends on VMX being enabled. > > Add two functions, tdx_enable() and tdx_cpu_enable(), to enable the TDX > module and enable TDX on local cpu respectively. For now tdx_enable() > is a placeholder. The TODO list will be pared down as functionality is > added. > > Export both tdx_cpu_enable() and tdx_enable() for KVM use. > > In tdx_enable() use a state machine protected by mutex to make sure the > initialization will only be done once, as tdx_enable() can be called > multiple times (i.e. KVM module can be reloaded) and may be called > concurrently by other kernel components in the future. > > The per-cpu initialization on each cpu can only be done once during the > module's life time. Use a per-cpu variable to track its status to make > sure it is only done once in tdx_cpu_enable(). > > Also, a SEAMCALL to do TDX module global initialization must be done > once on any logical cpu before any per-cpu initialization SEAMCALL. Do > it inside tdx_cpu_enable() too (if hasn't been done). > > tdx_enable() can potentially invoke SEAMCALLs on any online cpus. The > per-cpu initialization must be done before those SEAMCALLs are invoked > on some cpu. To keep things simple, in tdx_cpu_enable(), always do the > per-cpu initialization regardless of whether the TDX module has been > initialized or not. And in tdx_enable(), don't call tdx_cpu_enable() > but assume the caller has disabled CPU hotplug, done VMXON and > tdx_cpu_enable() on all online cpus before calling tdx_enable(). > > Signed-off-by: Kai Huang > --- > > v11 -> v12: > - Simplified TDX module global init and lp init status tracking (David). > - Added comment around try_init_module_global() for using > raw_spin_lock() (Dave). > - Added one sentence to changelog to explain why to expose tdx_enable() > and tdx_cpu_enable() (Dave). > - Simplifed comments around tdx_enable() and tdx_cpu_enable() to use > lockdep_assert_*() instead. (Dave) > - Removed redundent "TDX" in error message (Dave). > > v10 -> v11: > - Return -NODEV instead of -EINVAL when CONFIG_INTEL_TDX_HOST is off. > - Return the actual error code for tdx_enable() instead of -EINVAL. > - Added Isaku's Reviewed-by. > > v9 -> v10: > - Merged the patch to handle per-cpu initialization to this patch to > tell the story better. > - Changed how to handle the per-cpu initialization to only provide a > tdx_cpu_enable() function to let the user of TDX to do it when the > user wants to run TDX code on a certain cpu. > - Changed tdx_enable() to not call cpus_read_lock() explicitly, but > call lockdep_assert_cpus_held() to assume the caller has done that. > - Improved comments around tdx_enable() and tdx_cpu_enable(). > - Improved changelog to tell the story better accordingly. > > v8 -> v9: > - Removed detailed TODO list in the changelog (Dave). > - Added back steps to do module global initialization and per-cpu > initialization in the TODO list comment. > - Moved the 'enum tdx_module_status_t' from tdx.c to local tdx.h > > v7 -> v8: > - Refined changelog (Dave). > - Removed "all BIOS-enabled cpus" related code (Peter/Thomas/Dave). > - Add a "TODO list" comment in init_tdx_module() to list all steps of > initializing the TDX Module to tell the story (Dave). > - Made tdx_enable() unverisally return -EINVAL, and removed nonsense > comments (Dave). > - Simplified __tdx_enable() to only handle success or failure. > - TDX_MODULE_SHUTDOWN -> TDX_MODULE_ERROR > - Removed TDX_MODULE_NONE (not loaded) as it is not necessary. > - Improved comments (Dave). > - Pointed out 'tdx_module_status' is software thing (Dave). > > v6 -> v7: > - No change. > > v5 -> v6: > - Added code to set status to TDX_MODULE_NONE if TDX module is not > loaded (Chao) > - Added Chao's Reviewed-by. > - Improved comments around cpus_read_lock(). > > - v3->v5 (no feedback on v4): > - Removed the check that SEAMRR and TDX KeyID have been detected on > all present cpus. > - Removed tdx_detect(). > - Added num_online_cpus() to MADT-enabled CPUs check within the CPU > hotplug lock and return early with error message. > - Improved dmesg printing for TDX module detection and initialization. > > > --- > arch/x86/include/asm/tdx.h | 4 + > arch/x86/virt/vmx/tdx/tdx.c | 162 ++++++++++++++++++++++++++++++++++++ > arch/x86/virt/vmx/tdx/tdx.h | 13 +++ > 3 files changed, 179 insertions(+) > > diff --git a/arch/x86/include/asm/tdx.h b/arch/x86/include/asm/tdx.h > index 4dfe2e794411..d8226a50c58c 100644 > --- a/arch/x86/include/asm/tdx.h > +++ b/arch/x86/include/asm/tdx.h > @@ -97,8 +97,12 @@ static inline long tdx_kvm_hypercall(unsigned int nr, unsigned long p1, > > #ifdef CONFIG_INTEL_TDX_HOST > bool platform_tdx_enabled(void); > +int tdx_cpu_enable(void); > +int tdx_enable(void); > #else /* !CONFIG_INTEL_TDX_HOST */ > static inline bool platform_tdx_enabled(void) { return false; } > +static inline int tdx_cpu_enable(void) { return -ENODEV; } > +static inline int tdx_enable(void) { return -ENODEV; } > #endif /* CONFIG_INTEL_TDX_HOST */ > > #endif /* !__ASSEMBLY__ */ > diff --git a/arch/x86/virt/vmx/tdx/tdx.c b/arch/x86/virt/vmx/tdx/tdx.c > index 141d12376c4d..29ca18f66d61 100644 > --- a/arch/x86/virt/vmx/tdx/tdx.c > +++ b/arch/x86/virt/vmx/tdx/tdx.c > @@ -13,6 +13,10 @@ > #include > #include > #include > +#include > +#include > +#include > +#include > #include > #include > #include > @@ -23,6 +27,13 @@ static u32 tdx_global_keyid __ro_after_init; > static u32 tdx_guest_keyid_start __ro_after_init; > static u32 tdx_nr_guest_keyids __ro_after_init; > > +static bool tdx_global_initialized; > +static DEFINE_RAW_SPINLOCK(tdx_global_init_lock); > +static DEFINE_PER_CPU(bool, tdx_lp_initialized); > + > +static enum tdx_module_status_t tdx_module_status; Why can't you switch to a simple bool here as well? It's either initialized or uninitialized. If uninitialized and you get an error, leave it uninitialized. The next caller will try again and fail again. -- Cheers, David / dhildenb