From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 9910FC4332F for ; Wed, 13 Dec 2023 06:41:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:MIME-Version:Message-ID:Date:Subject:CC :To:From:Reply-To:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References: List-Owner; bh=KhTwnMNuZs5d8800Wa/75ZD9JxaezIcE+/g9C6+ve1M=; b=MEcW93JyR9OPbM 0QjZ0k/ur2ejYOGPzNHWGmuSGeuoCcgco4Mg1f2a9xyZnnVCHb+dTt3pzahwVcHnDd6Mg7plVksmS /iaJRfS7n6DxuOWvIS1aZ6pBuODkerHwQMWmYXmfxV4Ow05MaEJi9roRgpypKaKhPk/eFr87TwBLh gRTZiViW4j3OdZRHMjsieKMKOhW2AkDBkhqV1/RS77TW8KeI4xZP3yc9uA5SPzL8/l8Ofddxh+O/d ahllcJb7O2JJKTNU3+Y2FYiUOhixn5lkkh3fi2gEIe2EveGHn9lxVlpnWW0aVBs5WRS+B+Op99N3w YB+ArYcSk7iBPEjgC1cw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1rDIvX-00DoZh-1E; Wed, 13 Dec 2023 06:41:03 +0000 Received: from smtp-fw-52004.amazon.com ([52.119.213.154]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1rDIvT-00DoXJ-1Z for kexec@lists.infradead.org; Wed, 13 Dec 2023 06:41:01 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1702449659; x=1733985659; h=from:to:cc:subject:date:message-id:mime-version: content-transfer-encoding; bh=94amqqZKB2zJhADw4oYeciSDrXheAqykdeuVtMiJBwE=; b=DkWh8UCQeP5o4pKlwDxHnYbty728MtNtJORvab6ogOs0wMkoA45fy57D fkJTAGxGy1/Upb9E0zCfhzbBvtlhPZamhQcSa5El3dS2QO9oEd664R8LO wIpMyd7HQfrm6kMYj+oEuL4rllaVbQ6BZwNZMxbM4kWbsMnhNhrC11Gyr A=; X-IronPort-AV: E=Sophos;i="6.04,272,1695686400"; d="scan'208";a="171634859" Received: from iad12-co-svc-p1-lb1-vlan2.amazon.com (HELO email-inbound-relay-iad-1d-m6i4x-00fceed5.us-east-1.amazon.com) ([10.43.8.2]) by smtp-border-fw-52004.iad7.amazon.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Dec 2023 06:40:53 +0000 Received: from smtpout.prod.us-east-1.prod.farcaster.email.amazon.dev (iad7-ws-svc-p70-lb3-vlan3.iad.amazon.com [10.32.235.38]) by email-inbound-relay-iad-1d-m6i4x-00fceed5.us-east-1.amazon.com (Postfix) with ESMTPS id E9EAFA0B0C; Wed, 13 Dec 2023 06:40:48 +0000 (UTC) Received: from EX19MTAEUA002.ant.amazon.com [10.0.43.254:56758] by smtpin.naws.eu-west-1.prod.farcaster.email.amazon.dev [10.0.22.100:2525] with esmtp (Farcaster) id 9438de3e-ca1c-46b1-a8b0-23d75734eed4; Wed, 13 Dec 2023 06:40:47 +0000 (UTC) X-Farcaster-Flow-ID: 9438de3e-ca1c-46b1-a8b0-23d75734eed4 Received: from EX19D014EUC004.ant.amazon.com (10.252.51.182) by EX19MTAEUA002.ant.amazon.com (10.252.50.126) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Wed, 13 Dec 2023 06:40:47 +0000 Received: from u5d18b891348c5b.ant.amazon.com (10.146.13.114) by EX19D014EUC004.ant.amazon.com (10.252.51.182) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.1118.40; Wed, 13 Dec 2023 06:40:37 +0000 From: James Gowans To: , Eric Biederman , "Sean Christopherson" CC: , , Paolo Bonzini , Marc Zyngier , Arnd Bergmann , Tony Luck , Borislav Petkov , Thomas Gleixner , Ingo Molnar , Chen-Yu Tsai , Jernej Skrabec , Samuel Holland , "Pavel Machek" , Sebastian Reichel , Orson Zhai , Alexander Graf , "Jan H . Schoenherr" Subject: [PATCH] kexec: do syscore_shutdown() in kernel_kexec Date: Wed, 13 Dec 2023 08:40:04 +0200 Message-ID: <20231213064004.2419447-1-jgowans@amazon.com> X-Mailer: git-send-email 2.34.1 MIME-Version: 1.0 X-Originating-IP: [10.146.13.114] X-ClientProxiedBy: EX19D031UWC004.ant.amazon.com (10.13.139.246) To EX19D014EUC004.ant.amazon.com (10.252.51.182) X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20231212_224059_679192_B45FD223 X-CRM114-Status: GOOD ( 20.99 ) X-BeenThere: kexec@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "kexec" Errors-To: kexec-bounces+kexec=archiver.kernel.org@lists.infradead.org syscore_shutdown() runs driver and module callbacks to get the system into a state where it can be correctly shut down. In commit 6f389a8f1dd2 ("PM / reboot: call syscore_shutdown() after disable_nonboot_cpus()") syscore_shutdown() was removed from kernel_restart_prepare() and hence got (incorrectly?) removed from the kexec flow. This was innocuous until commit 6735150b6997 ("KVM: Use syscore_ops instead of reboot_notifier to hook restart/shutdown") changed the way that KVM registered its shutdown callbacks, switching from reboot notifiers to syscore_ops.shutdown. As syscore_shutdown() is missing from kexec, KVM's shutdown hook is not run and virtualisation is left enabled on the boot CPU which results in triple faults when switching to the new kernel on Intel x86 VT-x with VMXE enabled. Fix this by adding syscore_shutdown() to the kexec sequence. In terms of where to add it, it is being added after migrating the kexec task to the boot CPU, but before APs are shut down. It is not totally clear if this is the best place: in commit 6f389a8f1dd2 ("PM / reboot: call syscore_shutdown() after disable_nonboot_cpus()") it is stated that "syscore_ops operations should be carried with one CPU on-line and interrupts disabled." APs are only offlined later in machine_shutdown(), so this syscore_shutdown() is being run while APs are still online. This seems to be the correct place as it matches where syscore_shutdown() is run in the reboot and halt flows - they also run it before APs are shut down. The assumption is that the commit message in commit 6f389a8f1dd2 ("PM / reboot: call syscore_shutdown() after disable_nonboot_cpus()") is no longer valid. KVM has been discussed here as it is what broke loudly by not having syscore_shutdown() in kexec, but this change impacts more than just KVM; all drivers/modules which register a syscore_ops.shutdown callback will now be invoked in the kexec flow. Looking at some of them like x86 MCE it is probably more correct to also shut these down during kexec. Maintainers of all drivers which use syscore_ops.shutdown are added on CC for visibility. They are: arch/powerpc/platforms/cell/spu_base.c .shutdown = spu_shutdown, arch/x86/kernel/cpu/mce/core.c .shutdown = mce_syscore_shutdown, arch/x86/kernel/i8259.c .shutdown = i8259A_shutdown, drivers/irqchip/irq-i8259.c .shutdown = i8259A_shutdown, drivers/irqchip/irq-sun6i-r.c .shutdown = sun6i_r_intc_shutdown, drivers/leds/trigger/ledtrig-cpu.c .shutdown = ledtrig_cpu_syscore_shutdown, drivers/power/reset/sc27xx-poweroff.c .shutdown = sc27xx_poweroff_shutdown, kernel/irq/generic-chip.c .shutdown = irq_gc_shutdown, virt/kvm/kvm_main.c .shutdown = kvm_shutdown, This has been tested by doing a kexec on x86_64 and aarch64. Fixes: 6735150b6997 ("KVM: Use syscore_ops instead of reboot_notifier to hook restart/shutdown") Signed-off-by: James Gowans Cc: Eric Biederman Cc: Paolo Bonzini Cc: Sean Christopherson Cc: Marc Zyngier Cc: Arnd Bergmann Cc: Tony Luck Cc: Borislav Petkov Cc: Thomas Gleixner Cc: Ingo Molnar Cc: Chen-Yu Tsai Cc: Jernej Skrabec Cc: Samuel Holland Cc: Pavel Machek Cc: Sebastian Reichel Cc: Orson Zhai Cc: Alexander Graf Cc: Jan H. Schoenherr --- kernel/kexec_core.c | 1 + 1 file changed, 1 insertion(+) diff --git a/kernel/kexec_core.c b/kernel/kexec_core.c index be5642a4ec49..b926c4db8a91 100644 --- a/kernel/kexec_core.c +++ b/kernel/kexec_core.c @@ -1254,6 +1254,7 @@ int kernel_kexec(void) kexec_in_progress = true; kernel_restart_prepare("kexec reboot"); migrate_to_reboot_cpu(); + syscore_shutdown(); /* * migrate_to_reboot_cpu() disables CPU hotplug assuming that -- 2.34.1 _______________________________________________ kexec mailing list kexec@lists.infradead.org http://lists.infradead.org/mailman/listinfo/kexec