From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Thu, 10 Nov 2022 16:58:56 +0000 Subject: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > ? int cpu = smp_processor_id(); > > > ? struct cpuinfo_x86 *c = &cpu_data(cpu); > > > ? > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > ? > > > > Also, the logic of: > > > > !irqs_disabled() && cpu_active(cpu) > > > > is quite weird. > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > IRQ has already been enabled (see start_secondary()).? IIUC the WARN_ON() > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > check. > > > > So, a more reasonable check should be something like: > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > Or we can simply do: > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > ONLINE section). > > > > Btw above is purely based on code analysis, but I haven't done any test. > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > false. > > So WARN_ON(!irqs_disabled() && cpu_active(cpu)) looks reasonable. Sorry for the > noise. Just needed some time to connect the comment with the code. No worries, more than once while working through this code, I've considered setting up one of those evidence boards from the movies with string and push pins to help connect all the dots. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id D01A8C43219 for ; Thu, 10 Nov 2022 16:59:05 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 59E434BAF0; Thu, 10 Nov 2022 11:59:05 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id p9r4b6zOo2z7; Thu, 10 Nov 2022 11:59:04 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 4E8EF4BAC4; Thu, 10 Nov 2022 11:59:04 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 1E7844BA94 for ; Thu, 10 Nov 2022 11:59:03 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 3UYa6m2i5fjg for ; Thu, 10 Nov 2022 11:59:02 -0500 (EST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 0A18C4BABE for ; Thu, 10 Nov 2022 11:59:01 -0500 (EST) Received: by mail-pl1-f180.google.com with SMTP id v17so1900295plo.1 for ; Thu, 10 Nov 2022 08:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=LnpIznW50UbJYIR0PEtWeJYNJ+5rH6MIlZozMzFMTwSQvTcP33ucW9c9lQSevvLDXI QyB5+yXhMZUDNlkwdSbDFjM7JR9S8ttOodJeLQApsp/g1X2g7iaiRMpztbbtLirGmHXP Ok5MPlCZ4tZXNzdoN/5EaAFNU5i/kMEU+Pj6UbsvHwVNko95a9bIdjWXiNhQrkyCMZcF 1gbXxHeTac5BLcvPalQHTWFwjbFrO0re69/ZDM4/c7ATiGknmYZEnSteDE0eFOsuiUGv o02wNWrUZNQBtGle02WgOsPhE6Inb5/XeEtyenAcor9m9bnEppDYL8UdqJt6p1agDB2o ED2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=jCDJ+zh/+jX7wprZfuxpLAm5DnG48Lf4Glh0V4xPGv6iHHU1R6EkQAdrl5L3crKSsl a05tKU2rAyB8TMppcWpJMnoyvB5ihpZMiiHWGYdp6BtT1yRCifL7szfxB+NwfOvbwNYo OWa2MFq6fmrckAWykYCQaJLZuBRf87ORhx4cP+xadYEq2LAaAsnOvTcXkPQ9TtvCM+Pf FIx6etngbFgkB6kGH9eOhPlO4+qq/49SdGQbvYtCVUceGsBdGAfmifSGhbVoUsnX8CVk dxODArr6jrjmS28iV1SpIgXz5k2O/yVBd2ghgvsXfTlQdSUI4DO/fgjhmFVperMQT8nR MqUQ== X-Gm-Message-State: ANoB5pl6D4rgp78wNA7WEdU1bl4//yIVjwYqxxk5cIVrIIBVeOX9oVPS E/8B24gByM/5tMMIG0WqDD1EgQ== X-Google-Smtp-Source: AA0mqf4HLI0425WgRwXjVOczy7lGJ1ZoFRhJMikRvyU6kmBBe6o5W1MVmVmeziJeqm75+QzM+47Lew== X-Received: by 2002:a17:902:6aca:b0:188:736c:befa with SMTP id i10-20020a1709026aca00b00188736cbefamr1100928plt.8.1668099540938; Thu, 10 Nov 2022 08:59:00 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id n63-20020a17090a5ac500b00200461cfa99sm3302123pji.11.2022.11.10.08.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 08:59:00 -0800 (PST) Date: Thu, 10 Nov 2022 16:58:56 +0000 From: Sean Christopherson To: "Huang, Kai" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: "mjrosato@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "paul.walmsley@sifive.com" , "linux-mips@vger.kernel.org" , "mpe@ellerman.id.au" , "linux-riscv@lists.infradead.org" , "imbrenda@linux.ibm.com" , "kvmarm@lists.cs.columbia.edu" , "linux-s390@vger.kernel.org" , "frankja@linux.ibm.com" , "maz@kernel.org" , "chenhuacai@kernel.org" , "aleksandar.qemu.devel@gmail.com" , "borntraeger@linux.ibm.com" , "Gao, Chao" , "farman@linux.ibm.com" , "aou@eecs.berkeley.edu" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "kvmarm@lists.linux.dev" , "tglx@linutronix.de" , "linux-arm-kernel@lists.infradead.org" , "Yamahata, Isaku" , "farosas@linux.ibm.com" , "linux-kernel@vger.kernel.org" , "palmer@dabbelt.com" , "kvm-riscv@lists.infradead.org" , "pbonzini@redhat.com" , "vkuznets@redhat.com" , "linuxppc-dev@lists.ozlabs.org" X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > =A0 int cpu =3D smp_processor_id(); > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > =A0 > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > =A0 > > = > > Also, the logic of: > > = > > !irqs_disabled() && cpu_active(cpu) > > = > > is quite weird. > > = > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > = > > But this doesn't make sense anymore after we move to ONLINE section, in= which > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WARN_= ON() > > doesn't get exploded is purely because there's an additional cpu_active= (cpu) > > check. > > = > > So, a more reasonable check should be something like: > > = > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > = > > Or we can simply do: > > = > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > = > > (because I don't know whether it's possible IRQ can somehow get disable= d in > > ONLINE section). > > = > > Btw above is purely based on code analysis, but I haven't done any test. > = > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disab= led and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_acti= ve() is > false. > = > So WARN_ON(!irqs_disabled() && cpu_active(cpu)) looks reasonable. Sorry = for the > noise. Just needed some time to connect the comment with the code. No worries, more than once while working through this code, I've considered= setting up one of those evidence boards from the movies with string and push pins t= o help connect all the dots. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pj1-f44.google.com (mail-pj1-f44.google.com [209.85.216.44]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id B53521A601 for ; Thu, 10 Nov 2022 16:59:01 +0000 (UTC) Received: by mail-pj1-f44.google.com with SMTP id h14so2017650pjv.4 for ; Thu, 10 Nov 2022 08:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=LnpIznW50UbJYIR0PEtWeJYNJ+5rH6MIlZozMzFMTwSQvTcP33ucW9c9lQSevvLDXI QyB5+yXhMZUDNlkwdSbDFjM7JR9S8ttOodJeLQApsp/g1X2g7iaiRMpztbbtLirGmHXP Ok5MPlCZ4tZXNzdoN/5EaAFNU5i/kMEU+Pj6UbsvHwVNko95a9bIdjWXiNhQrkyCMZcF 1gbXxHeTac5BLcvPalQHTWFwjbFrO0re69/ZDM4/c7ATiGknmYZEnSteDE0eFOsuiUGv o02wNWrUZNQBtGle02WgOsPhE6Inb5/XeEtyenAcor9m9bnEppDYL8UdqJt6p1agDB2o ED2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=a8ejWX/lb/3VXsCabIOkVFnfZxfqxx4eQFhY6DjE5Yg6GS9RHWamnKqsGFtxMLgk2h L/lRle6/lPXJk7p5Nhw3ap2fXP40UZcj8f097l9qPGPpBonwbHvH4yIQe5mFvQqJ5jAj TSZm3fpb8zlxcniwlOyaa2SApKf68o3L/gjSElcjuvhRmuxZ/e/9CYbscH0mwUwoG5Hc WOwH5/LYPYbXNYJrUA8ZFiu31uZY+Csjz1Qr8apHbJc0fFVqLMUp1u5owh/16kJrBX5s Dc6auYGdzH4bADHlt6rJc54vb7zBUIV3HqcVGn+SApdGLKRMTmmkVdeJb9UAViU1q9w8 Y5HQ== X-Gm-Message-State: ANoB5pks7kTg/dNgk3dMMOeYE9+vk4dyEvAUgskSN+yKHu0hA6tIBiBE EkuiJckZ8Kr3No+uS0U9Dwg5VA== X-Google-Smtp-Source: AA0mqf4HLI0425WgRwXjVOczy7lGJ1ZoFRhJMikRvyU6kmBBe6o5W1MVmVmeziJeqm75+QzM+47Lew== X-Received: by 2002:a17:902:6aca:b0:188:736c:befa with SMTP id i10-20020a1709026aca00b00188736cbefamr1100928plt.8.1668099540938; Thu, 10 Nov 2022 08:59:00 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id n63-20020a17090a5ac500b00200461cfa99sm3302123pji.11.2022.11.10.08.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 08:59:00 -0800 (PST) Date: Thu, 10 Nov 2022 16:58:56 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Message-ID: <20221110165856.RmjL5sdrcPuFdjPxLKKwLlAso3qSEVQNtJ-pi2QlbEs@z> On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > >   int cpu = smp_processor_id(); > > >   struct cpuinfo_x86 *c = &cpu_data(cpu); > > >   > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > >   > > > > Also, the logic of: > > > > !irqs_disabled() && cpu_active(cpu) > > > > is quite weird. > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > IRQ has already been enabled (see start_secondary()).  IIUC the WARN_ON() > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > check. > > > > So, a more reasonable check should be something like: > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > Or we can simply do: > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > ONLINE section). > > > > Btw above is purely based on code analysis, but I haven't done any test. > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > false. > > So WARN_ON(!irqs_disabled() && cpu_active(cpu)) looks reasonable. Sorry for the > noise. Just needed some time to connect the comment with the code. No worries, more than once while working through this code, I've considered setting up one of those evidence boards from the movies with string and push pins to help connect all the dots. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6A1DEC4332F for ; Thu, 10 Nov 2022 16:59:15 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=eHGDYRCz8uagqB6ihDFTI4hQXI8gTQkKm9/ZFyT7K1M=; b=3XCbpb59esHZIs HUFteFhF4IHgrPACV/YSVYJOT7UR5U2ZyO5z4hKcjhS9Nx0aLdmkbVg4KaBl4bdO9ftquJoVHVhkz x5mUJrXvV1zuhHZrs5dYhPZ7p2KJzwH/m+eT07J0YY9qR+CRawqCWEVUdS/zI6S/JL1exaA3oHqRX rkv9WM/l4fFJK7mMk4ZK+8sH7AEtLm2oXl9YHO3iKZFNzHBF0ktlrl5ss0TXgdYfH4IwYPbTv2KBG jFwM70vhsN5wEJ9BufrDh5CLZ5pXXtf18QaZVDIxpU5F65g6S93i2KhZ0Mgd8/Tl240sIT7aQ6F4A f5s/b6ZGbHHcgdUqfoLA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otAtN-007PJz-UA; Thu, 10 Nov 2022 16:59:05 +0000 Received: from mail-pl1-x630.google.com ([2607:f8b0:4864:20::630]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otAtK-007PHD-Fc for linux-riscv@lists.infradead.org; Thu, 10 Nov 2022 16:59:03 +0000 Received: by mail-pl1-x630.google.com with SMTP id j12so1883449plj.5 for ; Thu, 10 Nov 2022 08:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=LnpIznW50UbJYIR0PEtWeJYNJ+5rH6MIlZozMzFMTwSQvTcP33ucW9c9lQSevvLDXI QyB5+yXhMZUDNlkwdSbDFjM7JR9S8ttOodJeLQApsp/g1X2g7iaiRMpztbbtLirGmHXP Ok5MPlCZ4tZXNzdoN/5EaAFNU5i/kMEU+Pj6UbsvHwVNko95a9bIdjWXiNhQrkyCMZcF 1gbXxHeTac5BLcvPalQHTWFwjbFrO0re69/ZDM4/c7ATiGknmYZEnSteDE0eFOsuiUGv o02wNWrUZNQBtGle02WgOsPhE6Inb5/XeEtyenAcor9m9bnEppDYL8UdqJt6p1agDB2o ED2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=bE7prkfuaaZ7dOFDegWAJoo5rTRNWRYLueORshzhcb9B7O4Nmho50R4CAGfhJTwmWw Vfs+Bxn39AT4BcCwH6eQ2hCovIBYx1s1H++tIGvRqSfRxHhtToXIQNgYMHs2GnbXKkFh +Doib0A1tnwemr84yHE5QoJG4Si9nqixxGD2ACIZYQJk7wRYQdHwGFfdkxURlrsmkLAA 75TtrJokCZ3hU8dHzroyj0B0n6BF8/3PiscIuRnx1eJ7g4gviBZgoM80Fy4qTvKDrXm0 wD6QNON7W7O84uh2YcX7HcbF1uwWHv0vz1MH7lImMwI31nyHE2tB0BJDgbyqVq5MCA+G k1gg== X-Gm-Message-State: ANoB5pmgWYMHbhUDOQCtBGHsX5M5+kVDsLHRr0p8RmT1iaLkYOslQi+B B+LbdzcVjim7WKIAaFMc88EUjA== X-Google-Smtp-Source: AA0mqf4HLI0425WgRwXjVOczy7lGJ1ZoFRhJMikRvyU6kmBBe6o5W1MVmVmeziJeqm75+QzM+47Lew== X-Received: by 2002:a17:902:6aca:b0:188:736c:befa with SMTP id i10-20020a1709026aca00b00188736cbefamr1100928plt.8.1668099540938; Thu, 10 Nov 2022 08:59:00 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id n63-20020a17090a5ac500b00200461cfa99sm3302123pji.11.2022.11.10.08.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 08:59:00 -0800 (PST) Date: Thu, 10 Nov 2022 16:58:56 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221110_085902_540867_9DF15D54 X-CRM114-Status: GOOD ( 23.09 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > =A0 int cpu =3D smp_processor_id(); > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > =A0 > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > =A0 > > = > > Also, the logic of: > > = > > !irqs_disabled() && cpu_active(cpu) > > = > > is quite weird. > > = > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > = > > But this doesn't make sense anymore after we move to ONLINE section, in= which > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WARN_= ON() > > doesn't get exploded is purely because there's an additional cpu_active= (cpu) > > check. > > = > > So, a more reasonable check should be something like: > > = > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > = > > Or we can simply do: > > = > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > = > > (because I don't know whether it's possible IRQ can somehow get disable= d in > > ONLINE section). > > = > > Btw above is purely based on code analysis, but I haven't done any test. > = > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disab= led and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_acti= ve() is > false. > = > So WARN_ON(!irqs_disabled() && cpu_active(cpu)) looks reasonable. Sorry = for the > noise. Just needed some time to connect the comment with the code. No worries, more than once while working through this code, I've considered= setting up one of those evidence boards from the movies with string and push pins t= o help connect all the dots. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 2D82EC43217 for ; Thu, 10 Nov 2022 16:59:59 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4N7SkP4PGnz3f3x for ; Fri, 11 Nov 2022 03:59:57 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=LnpIznW5; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2607:f8b0:4864:20::632; helo=mail-pl1-x632.google.com; envelope-from=seanjc@google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=LnpIznW5; dkim-atps=neutral Received: from mail-pl1-x632.google.com (mail-pl1-x632.google.com [IPv6:2607:f8b0:4864:20::632]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4N7SjN6ZHHz2x9J for ; Fri, 11 Nov 2022 03:59:03 +1100 (AEDT) Received: by mail-pl1-x632.google.com with SMTP id p21so1875643plr.7 for ; Thu, 10 Nov 2022 08:59:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=LnpIznW50UbJYIR0PEtWeJYNJ+5rH6MIlZozMzFMTwSQvTcP33ucW9c9lQSevvLDXI QyB5+yXhMZUDNlkwdSbDFjM7JR9S8ttOodJeLQApsp/g1X2g7iaiRMpztbbtLirGmHXP Ok5MPlCZ4tZXNzdoN/5EaAFNU5i/kMEU+Pj6UbsvHwVNko95a9bIdjWXiNhQrkyCMZcF 1gbXxHeTac5BLcvPalQHTWFwjbFrO0re69/ZDM4/c7ATiGknmYZEnSteDE0eFOsuiUGv o02wNWrUZNQBtGle02WgOsPhE6Inb5/XeEtyenAcor9m9bnEppDYL8UdqJt6p1agDB2o ED2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=3O/nAwpVIEvIl6obY4p7BgKCBLKYdRhd0e5HjazbiAQSG8Z4bppE4O+v/BXj8xGViL te7J0XXU8/FQN6KFZ/dG3Bbig0Dgb56GYRgDbwyz2PoLE6Kl6cRTz0VKjhOrMLxmsciF ld9XqTtfPluw9LP+Nhc8gChc22cfwn/QdeUsJMoQBTjTpXbqMIYHWzzapfpfhxljdrji 2HyapaOvrjcMQcRNzaClci62QJS1xycIfE++gYb+KLo+8Gt0Z7kiMH4A1Ao3LQhHSzCC 73KSKIY+ojLIyLY7kf0/jBDooyogloDC4TFKsAVb7Er5Jr0YZvyiFUrVZ3J687nHp0RZ fmjw== X-Gm-Message-State: ANoB5pkGc+dSwhBoBOYTLgv7FS5RHW68PXul+ios7i4jtt7q90c7DWrb Y1ZJhKFComCfqrJe7LJ5J2+L7A== X-Google-Smtp-Source: AA0mqf4HLI0425WgRwXjVOczy7lGJ1ZoFRhJMikRvyU6kmBBe6o5W1MVmVmeziJeqm75+QzM+47Lew== X-Received: by 2002:a17:902:6aca:b0:188:736c:befa with SMTP id i10-20020a1709026aca00b00188736cbefamr1100928plt.8.1668099540938; Thu, 10 Nov 2022 08:59:00 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id n63-20020a17090a5ac500b00200461cfa99sm3302123pji.11.2022.11.10.08.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 08:59:00 -0800 (PST) Date: Thu, 10 Nov 2022 16:58:56 +0000 From: Sean Christopherson To: "Huang, Kai" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "mjrosato@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "paul.walmsley@sifive.com" , "linux-mips@vger.kernel.org" , "linux-riscv@lists.infradead.org" , "imbrenda@linux.ibm.com" , "kvmarm@lists.cs.columbia.edu" , "linux-s390@vger.kernel.org" , "frankja@linux.ibm.com" , "maz@kernel.org" , "chenhuacai@kernel.org" , "aleksandar.qemu.devel@gmail.com" , "james.morse@arm.com" , "borntraeger@linux.ibm.com" , "Gao, Chao" , "farman@linux.ibm.com" , "aou@eecs.berkeley.edu" , "suzuki.poulose@arm.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "kvmarm@lists.linux.dev" , "tglx@linutronix.de" , "alexandru.elisei@arm.com" , "linux-arm-kernel@lists.infradead.org" , "Yamahata, Isaku" , "farosas@linux.ibm.com" , "linux-kernel@vger.kernel.org" , "oliver.upton@linux.dev" , "palmer@dabbelt.com" , "kvm-riscv@lists.infradead.org" , "anup@brainfault.org" , "pbonzini@redhat.com" , "vkuznets@redhat.com" , "linuxppc-dev@lists.ozlabs.org" Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > >   int cpu = smp_processor_id(); > > >   struct cpuinfo_x86 *c = &cpu_data(cpu); > > >   > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > >   > > > > Also, the logic of: > > > > !irqs_disabled() && cpu_active(cpu) > > > > is quite weird. > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > IRQ has already been enabled (see start_secondary()).  IIUC the WARN_ON() > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > check. > > > > So, a more reasonable check should be something like: > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > Or we can simply do: > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > ONLINE section). > > > > Btw above is purely based on code analysis, but I haven't done any test. > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > false. > > So WARN_ON(!irqs_disabled() && cpu_active(cpu)) looks reasonable. Sorry for the > noise. Just needed some time to connect the comment with the code. No worries, more than once while working through this code, I've considered setting up one of those evidence boards from the movies with string and push pins to help connect all the dots. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B1FE9C4332F for ; Thu, 10 Nov 2022 17:00:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=LTUVVN/rAx3gwNDzC3/WMq+Nj6+fLVjNVNVRC4KWvEk=; b=vKvf7PQFwWwYPj 6tttCKhqBKdxMPonxhP/16A4HrNJkeCzxhAAQ8q1VZ31ns1jQrhSuhoG+0JFpFH9uYUYgoveKDgIZ ZgL01afaOqyS/MV5z8IqWlSE1CKXlf8lCyN8nAMIoLuS93O2c0sDPKYE0/upqWQCpm21FwQ7LC9Gy EAMkNQTF5Ibe2/B+tGPTP7rONGC3H901JS8Lpr0v+t6P5vC0Dmas+wHIOhEMhlDiFfPquB7wbznaT vbBF36orTQpojieha85orTKdaiUpg4rrv6wM6NquwSPshAMtus/yYHotlv8lqo19u8JAH7/h+nVMQ xLHjb+1j8C+zcCTBO3DA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1otAtQ-007PLk-8p; Thu, 10 Nov 2022 16:59:08 +0000 Received: from mail-pj1-x102b.google.com ([2607:f8b0:4864:20::102b]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1otAtK-007PHE-Fc for linux-arm-kernel@lists.infradead.org; Thu, 10 Nov 2022 16:59:05 +0000 Received: by mail-pj1-x102b.google.com with SMTP id gw22so2021619pjb.3 for ; Thu, 10 Nov 2022 08:59:01 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=LnpIznW50UbJYIR0PEtWeJYNJ+5rH6MIlZozMzFMTwSQvTcP33ucW9c9lQSevvLDXI QyB5+yXhMZUDNlkwdSbDFjM7JR9S8ttOodJeLQApsp/g1X2g7iaiRMpztbbtLirGmHXP Ok5MPlCZ4tZXNzdoN/5EaAFNU5i/kMEU+Pj6UbsvHwVNko95a9bIdjWXiNhQrkyCMZcF 1gbXxHeTac5BLcvPalQHTWFwjbFrO0re69/ZDM4/c7ATiGknmYZEnSteDE0eFOsuiUGv o02wNWrUZNQBtGle02WgOsPhE6Inb5/XeEtyenAcor9m9bnEppDYL8UdqJt6p1agDB2o ED2w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=++qaNnNL5OJX09F4SmymJbSNGTe4YNGKf8knlNeZLIM=; b=R6+VkPq2ApL5+ckJDfQ9mhe9Sfq02nAieGt6BCZvkmEXX4FulfVxEQSnQY0Q3OnCCC Z41Th8gsflyJASMYHz3Pmo6U8gAGvf9DnjDy06lJCDZp6wItlLRmpss+U60VpJQj2FYE BJpdyTovy4V70gwFF8itqOySIaSdwiMlGDxpAd97ckgIfrz8oKDUCvWZFc9f69M3a4hJ UzDD7iXBI7w0er3O1pAUXIR32BdiT2rh4V/l7s2LfDlnOzjOafakXtu+K/NXQyC+yhXF Y3r0hXuwgsR0hs3cuKAN/zyl3XPp9fRmz5ffNbJVFUnRoJyFj2kfDx7r7mptX73vnmgS i4yQ== X-Gm-Message-State: ANoB5plLMvRxwjuJOjT/wUXFAYu1Ck5gVNrUbA8Zhk0aJ7RjNy543ngh 9i9pWSlU41FB4Iv2qjXpeV6YiQ== X-Google-Smtp-Source: AA0mqf4HLI0425WgRwXjVOczy7lGJ1ZoFRhJMikRvyU6kmBBe6o5W1MVmVmeziJeqm75+QzM+47Lew== X-Received: by 2002:a17:902:6aca:b0:188:736c:befa with SMTP id i10-20020a1709026aca00b00188736cbefamr1100928plt.8.1668099540938; Thu, 10 Nov 2022 08:59:00 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id n63-20020a17090a5ac500b00200461cfa99sm3302123pji.11.2022.11.10.08.59.00 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 10 Nov 2022 08:59:00 -0800 (PST) Date: Thu, 10 Nov 2022 16:58:56 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221110_085902_540256_BCB0A524 X-CRM114-Status: GOOD ( 24.67 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > =A0 int cpu =3D smp_processor_id(); > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > =A0 > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > =A0 > > = > > Also, the logic of: > > = > > !irqs_disabled() && cpu_active(cpu) > > = > > is quite weird. > > = > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > = > > But this doesn't make sense anymore after we move to ONLINE section, in= which > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WARN_= ON() > > doesn't get exploded is purely because there's an additional cpu_active= (cpu) > > check. > > = > > So, a more reasonable check should be something like: > > = > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > = > > Or we can simply do: > > = > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > = > > (because I don't know whether it's possible IRQ can somehow get disable= d in > > ONLINE section). > > = > > Btw above is purely based on code analysis, but I haven't done any test. > = > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disab= led and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_acti= ve() is > false. > = > So WARN_ON(!irqs_disabled() && cpu_active(cpu)) looks reasonable. Sorry = for the > noise. Just needed some time to connect the comment with the code. No worries, more than once while working through this code, I've considered= setting up one of those evidence boards from the movies with string and push pins t= o help connect all the dots. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel