From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DFAADC7EE23 for ; Tue, 23 May 2023 08:05:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:References :In-Reply-To:Subject:Cc:To:From:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=sw1XJ0hzqZ/Z1jgg5KwXg57AMMR7iDf1StXrIrGAnzk=; b=BVzLzsadripxvL LrX9ViLr+NOQce2IXpu7hTVsFaDeswxR+Y78v0zHciHCTwQTgSCNVvV1VWlI3W6RqI4KsAIm8Fe07 cKEhHitNFDonj3gTqTXJx/TjsrdJWPwwxUVm2en7o7WIi3tQm92tLvo5h2fXcwjhvcEP0SdYr775r h5NdmwxdI5sEETZIRm/zrjTjKypiOBj94zpP4rDXRj+R74JjAY+D3UE0EgYED0eWm4y29alhE2lxG bTZ3M6I6k/tPc7CbJYfspbo1sg8mnHC/WSMJSetUbeJGsfOEGGI4mG9OJsrenPDBpx7aM1p/ZXCoY L/dnwC6hVTlA5WstQDKQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1q1N0s-009Msv-2c; Tue, 23 May 2023 08:04:58 +0000 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1q1N0p-009Mrp-20 for linux-arm-kernel@lists.infradead.org; Tue, 23 May 2023 08:04:57 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1684829092; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=+spMR7GWVCywyQ8Vss3L/nhZ/nrGXvNU+qf7VC841P0=; b=iKNLadwQuLvoG3ZGJzn1kjQnZdYcCuPx7DkXzUTNQPaCjKIBKMuV1jpekw9L1Xymt8OzMo 8U6Vi4aKraZhS0mNPeynJLGyN7njTa8fBGZMCLdjMvXMkvotHxkxu32TPeSayYt4K+8We1 VFqrlGqkkrZ+1xxYwP7UySLB/mauWe4= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-378-Pl3FqFk0P-6f8vdcZBsx7g-1; Tue, 23 May 2023 04:04:51 -0400 X-MC-Unique: Pl3FqFk0P-6f8vdcZBsx7g-1 Received: by mail-wm1-f72.google.com with SMTP id 5b1f17b1804b1-3f603b8eb61so12638985e9.0 for ; Tue, 23 May 2023 01:04:50 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1684829090; x=1687421090; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=+spMR7GWVCywyQ8Vss3L/nhZ/nrGXvNU+qf7VC841P0=; b=YmtWZ44PTbHOJwgp4kaRIWp6lxn3uxHJznSH43BBuPMR1CaL2QcRrijW+hAlmHXjW6 OOb8kgzbphqgCS7PHOmhAL0QGp/wnL6oH6T+ZgkBzUgccPef1o1eKG9qz/dsUG90T+mb 3xbXCLu+Jh9D+yiEy426Gm7fC323Fuzw23kWjdXtG75OI1Qs+aoSw/MsE3T2dSuh0jgV O5B0+woJS+cMXIaTFaTks5XBNvhHOluUYzyx+AjFcr1XPr0SmZsiOPKB0R2Thb6v6KEE hkbohc+fTiNSraLokfX8XGJ3X3AdA0UvzzvR/8etRYJ126saifpDrAVr/HMD3YG/0uqZ VjBA== X-Gm-Message-State: AC+VfDzIvnz8R6rHzvPwxNd28n6/eAzfuEqZJnIsoEMZekh8+X2H+vrp X/Inb4/TiRDLs1gFaZeibv/MhxhTCEQu21/jmEXe8A981UQdq/APMRS9eSJpFokPqrLofqyqq97 cZM6ZNI0/NfHmk04xyCLBQ27vpN7F8kfAHbE= X-Received: by 2002:a1c:f611:0:b0:3f4:2e13:ccdc with SMTP id w17-20020a1cf611000000b003f42e13ccdcmr9214194wmc.0.1684829089896; Tue, 23 May 2023 01:04:49 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7dZ+uqenjEqCbZmwJMUh612CQFPX8E4+NvP9d6E10gJlbOWWzX9LXpbLljFvjphuDg6sVdlQ== X-Received: by 2002:a1c:f611:0:b0:3f4:2e13:ccdc with SMTP id w17-20020a1cf611000000b003f42e13ccdcmr9214164wmc.0.1684829089591; Tue, 23 May 2023 01:04:49 -0700 (PDT) Received: from fedora (g2.ign.cz. [91.219.240.8]) by smtp.gmail.com with ESMTPSA id v7-20020a7bcb47000000b003f080b2f9f4sm14233350wmj.27.2023.05.23.01.04.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 23 May 2023 01:04:49 -0700 (PDT) From: Vitaly Kuznetsov To: "Michael Kelley (LINUX)" Cc: KY Srinivasan , Haiyang Zhang , "wei.liu@kernel.org" , Dexuan Cui , "catalin.marinas@arm.com" , "will@kernel.org" , "tglx@linutronix.de" , "mingo@redhat.com" , "bp@alien8.de" , "dave.hansen@linux.intel.com" , "hpa@zytor.com" , "linux-kernel@vger.kernel.org" , "linux-hyperv@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "x86@kernel.org" , "kvm@vger.kernel.org" Subject: RE: [PATCH 1/2] x86/hyperv: Fix hyperv_pcpu_input_arg handling when CPUs go online/offline In-Reply-To: References: <1684506832-41392-1-git-send-email-mikelley@microsoft.com> <87o7mczqvu.fsf@redhat.com> Date: Tue, 23 May 2023 10:04:47 +0200 Message-ID: <87wn0zxylc.fsf@redhat.com> MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230523_010455_728513_C95AE1B4 X-CRM114-Status: GOOD ( 26.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org "Michael Kelley (LINUX)" writes: > From: Vitaly Kuznetsov Sent: Monday, May 22, 2023 1:56 AM >> >> Michael Kelley writes: >> > > [snip] > >> > diff --git a/include/linux/cpuhotplug.h b/include/linux/cpuhotplug.h >> > index 0f1001d..696004a 100644 >> > --- a/include/linux/cpuhotplug.h >> > +++ b/include/linux/cpuhotplug.h >> > @@ -201,6 +201,7 @@ enum cpuhp_state { >> > /* Online section invoked on the hotplugged CPU from the hotplug thread */ >> > CPUHP_AP_ONLINE_IDLE, >> > CPUHP_AP_KVM_ONLINE, >> > + CPUHP_AP_HYPERV_ONLINE, >> >> (Cc: KVM) >> >> I would very much prefer we swap the order with KVM here: hv_cpu_init() >> allocates and sets vCPU's VP assist page which is used by KVM on >> CPUHP_AP_KVM_ONLINE: >> >> kvm_online_cpu() -> __hardware_enable_nolock() -> >> kvm_arch_hardware_enable() -> vmx_hardware_enable(): >> >> /* >> * This can happen if we hot-added a CPU but failed to allocate >> * VP assist page for it. >> */ >> if (kvm_is_using_evmcs() && !hv_get_vp_assist_page(cpu)) >> return -EFAULT; >> >> With the change, this is likely broken. >> >> FWIF, KVM also needs current vCPU's VP index (also set by hv_cpu_init()) >> through __kvm_x86_vendor_init() -> set_hv_tscchange_cb() call chain but >> this happens upon KVM module load so CPU hotplug ordering should not >> matter as this always happens on a CPU which is already online. >> >> Generally, as 'KVM on Hyper-V' is a supported scenario, doing Hyper-V >> init before KVM's (and vice versa on teardown) makes sense. >> >> > CPUHP_AP_SCHED_WAIT_EMPTY, >> > CPUHP_AP_SMPBOOT_THREADS, >> > CPUHP_AP_X86_VDSO_VMA_ONLINE, > > I have no objection to putting CPUHP_AP_HYPERV_ONLINE first. I did > not give any consideration to a possible dependency between the two. :-( > But note that in current code, hv_cpu_init() is running on the > CPUHP_AP_ONLINE_DYN state, which is also after KVM. So this patch > doesn't change the order w.r.t. KVM and the VP assist page. Are things > already broken for KVM, or is something else happening that makes it > work anyway? This looks like a currently present bug indeed so I had to refresh my memory. KVM's CPUHP_AP_KVM_STARTIN is registered with cpuhp_setup_state_nocalls() which means that kvm_starting_cpu() is not called for all currently present CPUs. Moreover, kvm_init() is called when KVM vendor module (e.g. kvm_intel) is loaded and as KVM is normally built as module, this happens much later than Hyper-V's hyperv_init(). vmx_hardware_enable() is actually called from hardware_enable_all() which happens when the first KVM VM is created. This all changes when a CPU is hotplugged. The order CPUHP_AP_* from cpuhp_state is respected and KVM's kvm_starting_cpu() is called _before_ Hyper-V's hv_cpu_init() even before your patch. We don't see the bug just because Hyper-V doesn't(?) support CPU hotplug. Just sending a CPU offline with e.g. "echo 0 > /sys/devices/system/cpu/cpuX/online" is not the same as once allocated, VP assist page persists for all non-root Hyper-V partitions. I don't know if KVM is supported for Hyper-V root partitions but in case it is, we may have a problem. Let's put CPUHP_AP_HYPERV_ONLINE before KVM's CPUHP_AP_KVM_ONLINE explicitly so CPU hotplug scenario is handled correctly, even if current Hyper-V versions don't support it. -- Vitaly _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel