From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2EE55C64ED6 for ; Mon, 13 Feb 2023 22:28:52 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A4D92280001; Mon, 13 Feb 2023 17:28:51 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9D6F06B0074; Mon, 13 Feb 2023 17:28:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8500C280001; Mon, 13 Feb 2023 17:28:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 7162B6B0072 for ; Mon, 13 Feb 2023 17:28:51 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 18C3C1A0C75 for ; Mon, 13 Feb 2023 22:28:51 +0000 (UTC) X-FDA: 80463709662.24.3FCD814 Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by imf10.hostedemail.com (Postfix) with ESMTP id 42B0DC0016 for ; Mon, 13 Feb 2023 22:28:48 +0000 (UTC) Authentication-Results: imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ctlVE+8n; spf=pass (imf10.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1676327329; a=rsa-sha256; cv=none; b=OjVYBM4gur2XxJ1Yl8oocXKd1OkByasxwA+6K+bH2e3lpcP30X09hartoPUsWBIs2Fg31X tPIizgQJAxa/H9wv5NxVz1dQbAcXhtWdECNPwr+66I6Pn00i7yP518RNRzn3DkWHgkbFVN TUEUc3aVgNYuPB5JnDXa5mV7LwhvCUQ= ARC-Authentication-Results: i=1; imf10.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=ctlVE+8n; spf=pass (imf10.hostedemail.com: domain of dave.hansen@intel.com designates 192.55.52.88 as permitted sender) smtp.mailfrom=dave.hansen@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1676327329; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=BPsaNQ6W0FlXMUfv5wA5EAz6Pk4Uxbb4mwixaldyCL8=; b=GywIeuen9VKB79XaSlmPrKFHV5Wzym507OMQhlKDsb6MqwoLFoouQeymjus+QC1xsNq+2Z RhSOzLK/lFVA5KADLXeMVV4ILlB1n4fCdQrHj8oJXcBk4hMNwZaoECqZNkeixlqdtw7Tks ptZoLUazbkq9lgSzOWCGiDoT2gU/PrU= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1676327328; x=1707863328; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=314vpw90/2DwHi9JOkrC++51z0wYP6vtdKNmpuIX6Gc=; b=ctlVE+8nxj214mG7mm1SVSi57bgfkd97Aee+8ed1aS9nPPfrjs8+SS4J 4Mukni4DNz/DM7c9dQETOYII9BX0OFNfsm3n98wiuL8SYUAMbGzL2zDoh utSEiL8SMv7TlUkg4KgeTeeyOxvmBnXCpiPCJwbipHfMsQG+d5+Ui9i57 hORVvPjrk6DRenXDFP8YLACNXbQbKfMnvRt1TLohKlxc9Fz6YZlSGOukE F4tk44rSZikdWq2CbMGBxMhcR5oVehCx4vBbSRZbLlTeaNN/C6rtm4APq Yw+OYWnSQ9g/wzOrqJO9QCIxmec7wefJvm1rkRBZOdkfLUCGXujtVuNaa A==; X-IronPort-AV: E=McAfee;i="6500,9779,10620"; a="358421210" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="358421210" Received: from fmsmga004.fm.intel.com ([10.253.24.48]) by fmsmga101.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 14:28:46 -0800 X-IronPort-AV: E=McAfee;i="6500,9779,10620"; a="737674268" X-IronPort-AV: E=Sophos;i="5.97,294,1669104000"; d="scan'208";a="737674268" Received: from mlswanso-mobl.amr.corp.intel.com (HELO [10.251.26.232]) ([10.251.26.232]) by fmsmga004-auth.fm.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Feb 2023 14:28:44 -0800 Message-ID: <2d7141b1-1d76-4e67-60d2-471a524c372e@intel.com> Date: Mon, 13 Feb 2023 14:28:43 -0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.1 Subject: Re: [PATCH v9 07/18] x86/virt/tdx: Do TDX module per-cpu initialization Content-Language: en-US To: "Huang, Kai" , "linux-kernel@vger.kernel.org" , "kvm@vger.kernel.org" Cc: "linux-mm@kvack.org" , "peterz@infradead.org" , "tglx@linutronix.de" , "Christopherson,, Sean" , "pbonzini@redhat.com" , "Williams, Dan J" , "Wysocki, Rafael J" , "kirill.shutemov@linux.intel.com" , "Huang, Ying" , "Chatre, Reinette" , "Brown, Len" , "Luck, Tony" , "ak@linux.intel.com" , "Yamahata, Isaku" , "Gao, Chao" , "sathyanarayanan.kuppuswamy@linux.intel.com" , "david@redhat.com" , "bagasdotme@gmail.com" , "Shahar, Sagi" , "imammedo@redhat.com" References: <557c526a1190903d11d67c4e2c76e01f67f6eb15.1676286526.git.kai.huang@intel.com> <2d9172c5-e1e7-bf94-c52b-0e9bc5b5b319@intel.com> From: Dave Hansen In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 7bit X-Rspam-User: X-Rspamd-Queue-Id: 42B0DC0016 X-Rspamd-Server: rspam01 X-Stat-Signature: 7r577ytou8osqzksn6kowio5wpthb67m X-HE-Tag: 1676327328-830581 X-HE-Meta: U2FsdGVkX1+QFdc+63X7dRj4hYEkkfteGmMcpiVfTjpifUzrOqms9p86ZHXNNkkTlBAtRdtnGksRSfGxirP2NnrKVnrISm7BES/w5d7vbcQuCLzd9eWQvqYdUeoc3k0G5+OcdbF2F0afCcqm2UB78vGgdn3wewTsJ6m0OEVPBktKK6Srk3VU4+cVy5ySfRJFQV/p/tDkA4paDwycwpPA8lIhNIYG8/IfNSmNsARtlFkTxO3KlYr8HVVL+6nnuMqDaPKulP4V7F/j48yeR32UTWCIQ4DlUwjfIIOvFC4141flsNsAJpIHD5K7tE7QRPtQFKfAD3l4r1q2l6AeNMmJcZPbJO1DQhub74qQKMueYYMZUQ0lSxTCWlSgKSo4Rv4o8dgRy+6ax4TngplxMrFy+e/hY+lxedxLdVB+hpc2ieR3+GcPzm3qPdFa5JtN4tW035P3jBANZvhKUTJ01zOvwgfhs15qT/NQpKfB2WdXRDMmKkUscwmdtmNVW+TYsC2txy0q74FE4SqZTGLqiH20sYcMXzb2+YmfbZjN7zo9E4gif/2AdMiC3rkEM5bgs7JXp+pffUymxAuIwmz6bZ5nCjmbStiJv3gDEOxyyWuV7+8JBWUerpm9Sn6zaQdyZqlrJPqoIxVmvAEbX5g1+TErqHR980QyJU9cPDMZANJCXaPKcPX9lL2/7Oft2axDnaQJupfY1Ped5D7W4t12v7qw+wPhnZj534ZxlK2e1pwThFpdwILmlowryO0GoOUCNSu4OYyO4ddnBnZzBvzD4aqgU/Y6vfmQJng0p8XVl0NBLlC9jTp+Sz5EhQZ1/ryLKOanFX1Zp3xb7m0519CLPN34dIOS0Ro4BW/EKZp16pbx0tEE1ejozWgkhXPl0Txa4RQuK3lplOrQ//aAF6aaXTH5g119koyVg7MbHK2frL3uB81rMdJncJWnis7vs76jh0b4qssb+4x/Ng+iL311pZ5 krtWzTq3 Fo4VQM+1+vXwwYHRmM4NzjPHMV5gtzKk2lVC0tcCOJET7x2u/DMpiSKklmunM/1vH5zwtLaEOHskgBbYGXb8f6qZJwLcyBqeKZXfjmmdrI3/MFjg7r70g7JdUs5+tQ+KiGqKC3zHJ5lb2eOhVghg3JYX0sb/jESQRDoTw X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2/13/23 13:13, Huang, Kai wrote: > Perhaps I didn't explain clearly in the comment. Below is the updated one: > > /* > * The previous call of __tdx_enable() may only have > * initialized part of present cpus during module > * initialization, and new cpus may have become online > * since then w/o doing per-cpu initialization. > * > * For example, a new CPU can become online when KVM is > * unloaded, in which case tdx_cpu_enable() is not called since > * KVM's CPU online callback has been removed. > * > * To make sure all online cpus are TDX-runnable, always > * do per-cpu initialization for all online cpus here > * even the module has been initialized. > */ This is voodoo. I want a TDX-specific hotplug CPU handler. Period. Please make that happen. Put that code in this patch. That handler should: 1. Run after the KVM handler (if present) 2. See if VMX is on 3. If VMX is on: 3a. Run smp_func_module_lp_init(), else 3b. Mark the CPU as needing smp_func_module_lp_init() Then, in the 'case TDX_MODULE_INITIALIZED:', you call a function to iterate over the cpumask that was generated in 3b. That makes the handoff *EXPLICIT*. You know exactly which CPUs need what done to them. A CPU hotplug either explicitly involves doing the work to make TDX work on the CPU, or explicitly defers the work to a specific later time in a specific later piece of code.