From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Tue, 15 Nov 2022 20:16:04 +0000 Subject: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > ? int cpu = smp_processor_id(); > > > ? struct cpuinfo_x86 *c = &cpu_data(cpu); > > > ? > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > ? > > > > Also, the logic of: > > > > !irqs_disabled() && cpu_active(cpu) > > > > is quite weird. > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > IRQ has already been enabled (see start_secondary()).? IIUC the WARN_ON() > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > check. > > > > So, a more reasonable check should be something like: > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > Or we can simply do: > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > ONLINE section). > > > > Btw above is purely based on code analysis, but I haven't done any test. > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > false. Actually, you're right (and wrong). You're right in that the WARN is flawed. And the reason for that is because you're wrong about the hotplug case. In this version of things, the compatibility checks are routed through hardware enabling, i.e. this flow is used only when loading KVM. This helper should only be called via SMP function call, which means that IRQs should always be disabled. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4B305C43219 for ; Tue, 15 Nov 2022 20:16:13 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id B426E4B890; Tue, 15 Nov 2022 15:16:12 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 6ARDQMkGdTP3; Tue, 15 Nov 2022 15:16:11 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 7865E4B8BA; Tue, 15 Nov 2022 15:16:11 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 817A14B8B8 for ; Tue, 15 Nov 2022 15:16:10 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id RiawIdSxApWI for ; Tue, 15 Nov 2022 15:16:09 -0500 (EST) Received: from mail-pl1-f180.google.com (mail-pl1-f180.google.com [209.85.214.180]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 498724B890 for ; Tue, 15 Nov 2022 15:16:09 -0500 (EST) Received: by mail-pl1-f180.google.com with SMTP id c2so14148678plz.11 for ; Tue, 15 Nov 2022 12:16:09 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=CuhvpEh+MwcQ8DdyO2LHPKc75+Mh4Tt/SrTxun8SjsbCLbKA/uujVn75/hrSWRbq3P pp681Bo0ABWoVlpW4KZ12uslMC8XMyFowDj24K4rvmpM/9YujrW27P9Vv5MLxt+ZIfOV Jm6zqwFB/7ZHai0ByGJfyD247/5LpCxHhMpkaFu377RaNtmMs+gPLXFcf0G1yQbvs+Rs EYu+becUwCm5WaK54RGM6rYBIysavORuhi+fV9JxKIFpQITo1Jz1JNtjUgO7b5R2U1/e xEGMqJLbi5RxqShnMYvwl9BbYi//M2ZHV0GVHxnWgbMAYD9A/uq/G/wppECfL4cNv6ol hTbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=3XyLmzI69rwfL5iGP4PYEIrVg6qDSO2YjAQmKoeo8UkGO9i63OeZHXv7QDIbTyAZ9h +pvDOFYlf61de9nSA8LzWX2RLyOQQ2Q9YigJt3kzNFt1XWwssbec2MxslmspGFELfPuG tmDJSs8Ljf+XUGS+mK/RiACJfdyDP+T7IZpGsUX9ge4NStBo+vNpXxEU3bAKemiF5jm7 u8yEhmgMcAHAM7IK9OOVJUTRDJHVHEWZkNSi3xI0l6ik1+rnFRyZahP8ww2GZsfVswKC CgxdjuhQ8NY4eIHWTd08Nlxr/dMfQg8IOpjeK/Gz39LG1q6WBtLgyAMWJzCm/pWVQVGx QmiA== X-Gm-Message-State: ANoB5pkMnOOAaE+VM18dkSHVh/IZRbZlqpKHZOy/dUxQ+zTpxEFz1vjZ dZs8FM2K+vz/RxoeQOIKTbZJGw== X-Google-Smtp-Source: AA0mqf58/qb3pHX8jXG9rQwsI0kWr5CgUXW3YysHviPdV0VVEN3OAELl4gQs2//98Y2csv6jhLKLCw== X-Received: by 2002:a17:902:e8d5:b0:181:6c64:6dd3 with SMTP id v21-20020a170902e8d500b001816c646dd3mr5671131plg.123.1668543368117; Tue, 15 Nov 2022 12:16:08 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id d62-20020a623641000000b0056c08c87196sm9173979pfa.48.2022.11.15.12.16.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:16:07 -0800 (PST) Date: Tue, 15 Nov 2022 20:16:04 +0000 From: Sean Christopherson To: "Huang, Kai" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: "mjrosato@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "paul.walmsley@sifive.com" , "linux-mips@vger.kernel.org" , "mpe@ellerman.id.au" , "linux-riscv@lists.infradead.org" , "imbrenda@linux.ibm.com" , "kvmarm@lists.cs.columbia.edu" , "linux-s390@vger.kernel.org" , "frankja@linux.ibm.com" , "maz@kernel.org" , "chenhuacai@kernel.org" , "aleksandar.qemu.devel@gmail.com" , "borntraeger@linux.ibm.com" , "Gao, Chao" , "farman@linux.ibm.com" , "aou@eecs.berkeley.edu" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "kvmarm@lists.linux.dev" , "tglx@linutronix.de" , "linux-arm-kernel@lists.infradead.org" , "Yamahata, Isaku" , "farosas@linux.ibm.com" , "linux-kernel@vger.kernel.org" , "palmer@dabbelt.com" , "kvm-riscv@lists.infradead.org" , "pbonzini@redhat.com" , "vkuznets@redhat.com" , "linuxppc-dev@lists.ozlabs.org" X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > =A0 int cpu =3D smp_processor_id(); > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > =A0 > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > =A0 > > = > > Also, the logic of: > > = > > !irqs_disabled() && cpu_active(cpu) > > = > > is quite weird. > > = > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > = > > But this doesn't make sense anymore after we move to ONLINE section, in= which > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WARN_= ON() > > doesn't get exploded is purely because there's an additional cpu_active= (cpu) > > check. > > = > > So, a more reasonable check should be something like: > > = > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > = > > Or we can simply do: > > = > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > = > > (because I don't know whether it's possible IRQ can somehow get disable= d in > > ONLINE section). > > = > > Btw above is purely based on code analysis, but I haven't done any test. > = > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disab= led and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_acti= ve() is > false. Actually, you're right (and wrong). You're right in that the WARN is flawe= d. And the reason for that is because you're wrong about the hotplug case. In thi= s version of things, the compatibility checks are routed through hardware enabling, i= .e. this flow is used only when loading KVM. This helper should only be called via = SMP function call, which means that IRQs should always be disabled. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id C6BE48AAD for ; Tue, 15 Nov 2022 20:16:08 +0000 (UTC) Received: by mail-pl1-f171.google.com with SMTP id io19so14162414plb.8 for ; Tue, 15 Nov 2022 12:16:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=CuhvpEh+MwcQ8DdyO2LHPKc75+Mh4Tt/SrTxun8SjsbCLbKA/uujVn75/hrSWRbq3P pp681Bo0ABWoVlpW4KZ12uslMC8XMyFowDj24K4rvmpM/9YujrW27P9Vv5MLxt+ZIfOV Jm6zqwFB/7ZHai0ByGJfyD247/5LpCxHhMpkaFu377RaNtmMs+gPLXFcf0G1yQbvs+Rs EYu+becUwCm5WaK54RGM6rYBIysavORuhi+fV9JxKIFpQITo1Jz1JNtjUgO7b5R2U1/e xEGMqJLbi5RxqShnMYvwl9BbYi//M2ZHV0GVHxnWgbMAYD9A/uq/G/wppECfL4cNv6ol hTbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=b6X6ULbAZU7R9rTt9zrFZ+BR40QyRrcM0cmynJx3G9q415qSgq9tJixA69NmrufPzB cXytpGjhutcxs8EEE3aAS22qWNsHcRO11MgqLQg3k1gfsHEe6ExQ0kDdepEeKVheC0ue r18IIWsOfQ9JtCG+Ue55rZvJ5b9Te18vXgvJu7S3cx1pJeoSVeSspB1K9SaL/mvARl6c J+ipWrlCgbCU44M9fw8nEKuRXBoCP9UQXh746zYzp0zf+2RhZ+xfoYR/dDgHZ+vs9xUt pAgbqlHjQ5nwIXjTE8H6gq6oedMIssxhDBFoToCHfAXm+QDrwvBgTnQkwSWZ2H2ES+hV 57vw== X-Gm-Message-State: ANoB5plk0eGyc2ByYpe02SQ2QeajgmgkoMWLWa8Yv69PO6QlnOkymgxE NUJ6kD/WJBHdm3SqxLbnGviMLw== X-Google-Smtp-Source: AA0mqf58/qb3pHX8jXG9rQwsI0kWr5CgUXW3YysHviPdV0VVEN3OAELl4gQs2//98Y2csv6jhLKLCw== X-Received: by 2002:a17:902:e8d5:b0:181:6c64:6dd3 with SMTP id v21-20020a170902e8d500b001816c646dd3mr5671131plg.123.1668543368117; Tue, 15 Nov 2022 12:16:08 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id d62-20020a623641000000b0056c08c87196sm9173979pfa.48.2022.11.15.12.16.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:16:07 -0800 (PST) Date: Tue, 15 Nov 2022 20:16:04 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Message-ID: <20221115201604.LlNak4B6HIUVbbkMtAs_S6Mdnp3rMc-4SBadDn8KssE@z> On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > >   int cpu = smp_processor_id(); > > >   struct cpuinfo_x86 *c = &cpu_data(cpu); > > >   > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > >   > > > > Also, the logic of: > > > > !irqs_disabled() && cpu_active(cpu) > > > > is quite weird. > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > IRQ has already been enabled (see start_secondary()).  IIUC the WARN_ON() > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > check. > > > > So, a more reasonable check should be something like: > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > Or we can simply do: > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > ONLINE section). > > > > Btw above is purely based on code analysis, but I haven't done any test. > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > false. Actually, you're right (and wrong). You're right in that the WARN is flawed. And the reason for that is because you're wrong about the hotplug case. In this version of things, the compatibility checks are routed through hardware enabling, i.e. this flow is used only when loading KVM. This helper should only be called via SMP function call, which means that IRQs should always be disabled. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 6D8A3C433FE for ; Tue, 15 Nov 2022 20:16:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=yaKBWyap2AGPlrEseGmeAQFqUDJJSjTMgz5CDjbLQhE=; b=1AeCsUyz2AZpWD 0aLaXu5UH/Nf/ioVPIwkxxVvsCgsM1nbLiSuB0CEEhFqkKb9QcxBCRWtRCSeXURYLN/RYW/h2o1h1 UmvS20rp0bfjA3l1SnYXM+EWqt7xWN42mfwLhHu+e/fYIlHLd1NeiCaxgx6VaQu+8r+6YKKuLPmYG P7j9xEsT8eNAyn9gaAvv3SGP38mxetYKe0RseS/T+vwRhyH6e4SpjOKIiVgquX4OZ7eOQ5AaBn6C/ ojxWpinzHEj+kYTcNLmhl51ezX2wzZBgSTQsz9tG43sC8x3lTirrUBHCHAndqg57YvBLLbUUn67SA npN6U3gpdx6iK3ION4ww==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2M5-00EVwG-3l; Tue, 15 Nov 2022 20:16:25 +0000 Received: from mail-pl1-x62c.google.com ([2607:f8b0:4864:20::62c]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2Lu-00EVsa-8m for linux-riscv@lists.infradead.org; Tue, 15 Nov 2022 20:16:16 +0000 Received: by mail-pl1-x62c.google.com with SMTP id d20so14152804plr.10 for ; Tue, 15 Nov 2022 12:16:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=CuhvpEh+MwcQ8DdyO2LHPKc75+Mh4Tt/SrTxun8SjsbCLbKA/uujVn75/hrSWRbq3P pp681Bo0ABWoVlpW4KZ12uslMC8XMyFowDj24K4rvmpM/9YujrW27P9Vv5MLxt+ZIfOV Jm6zqwFB/7ZHai0ByGJfyD247/5LpCxHhMpkaFu377RaNtmMs+gPLXFcf0G1yQbvs+Rs EYu+becUwCm5WaK54RGM6rYBIysavORuhi+fV9JxKIFpQITo1Jz1JNtjUgO7b5R2U1/e xEGMqJLbi5RxqShnMYvwl9BbYi//M2ZHV0GVHxnWgbMAYD9A/uq/G/wppECfL4cNv6ol hTbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=m1lggBhHYoVVn43dihWLYv4LUhqyhdYwKVIhWq+n/EYiQ5L4lI/U4GJWLoWNeDWf4t ydWz0tYIbrm4t+JiEIkzrw8CncB6HjnW0HCwaNUE+W5eZ3CdgT9hHYmRdHmLYBa/P5XB r/cD6MfZBWWCGra7PiUNoAoG5vo7mIeBWdToLzXxyZPOQjCDUdwRhq76aw7OjlS8gFFv UEieMq1R6Cv+zv4jt0l2M/lkzYal6tsMlV7x6dJa3NJuY+6bnWCbOuXyDAGWpt9i8O5r sAqVOo0lsrUPqnp2aDhMvqSIvrG6oQkcfjLE3gI4ewlW6voq/tFqQW8xbnj3Rvfdz12U /+XQ== X-Gm-Message-State: ANoB5pniN9bI5OHRzHQom+v9+NMcJAYg3EVDDRSK+EvxxU2BBJ+ZTpcS qNQcA8TOZRXPtU1g6MNhHOd3oA== X-Google-Smtp-Source: AA0mqf58/qb3pHX8jXG9rQwsI0kWr5CgUXW3YysHviPdV0VVEN3OAELl4gQs2//98Y2csv6jhLKLCw== X-Received: by 2002:a17:902:e8d5:b0:181:6c64:6dd3 with SMTP id v21-20020a170902e8d500b001816c646dd3mr5671131plg.123.1668543368117; Tue, 15 Nov 2022 12:16:08 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id d62-20020a623641000000b0056c08c87196sm9173979pfa.48.2022.11.15.12.16.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:16:07 -0800 (PST) Date: Tue, 15 Nov 2022 20:16:04 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221115_121614_319269_927E638C X-CRM114-Status: GOOD ( 23.74 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > =A0 int cpu =3D smp_processor_id(); > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > =A0 > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > =A0 > > = > > Also, the logic of: > > = > > !irqs_disabled() && cpu_active(cpu) > > = > > is quite weird. > > = > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > = > > But this doesn't make sense anymore after we move to ONLINE section, in= which > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WARN_= ON() > > doesn't get exploded is purely because there's an additional cpu_active= (cpu) > > check. > > = > > So, a more reasonable check should be something like: > > = > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > = > > Or we can simply do: > > = > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > = > > (because I don't know whether it's possible IRQ can somehow get disable= d in > > ONLINE section). > > = > > Btw above is purely based on code analysis, but I haven't done any test. > = > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disab= led and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_acti= ve() is > false. Actually, you're right (and wrong). You're right in that the WARN is flawe= d. And the reason for that is because you're wrong about the hotplug case. In thi= s version of things, the compatibility checks are routed through hardware enabling, i= .e. this flow is used only when loading KVM. This helper should only be called via = SMP function call, which means that IRQs should always be disabled. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 1B6F6C4332F for ; Tue, 15 Nov 2022 20:17:11 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4NBcsd3g4Dz3cT4 for ; Wed, 16 Nov 2022 07:17:09 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=CuhvpEh+; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2607:f8b0:4864:20::102f; helo=mail-pj1-x102f.google.com; envelope-from=seanjc@google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=CuhvpEh+; dkim-atps=neutral Received: from mail-pj1-x102f.google.com (mail-pj1-x102f.google.com [IPv6:2607:f8b0:4864:20::102f]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4NBcrY2l8mz3bXR for ; Wed, 16 Nov 2022 07:16:12 +1100 (AEDT) Received: by mail-pj1-x102f.google.com with SMTP id d13-20020a17090a3b0d00b00213519dfe4aso14906082pjc.2 for ; Tue, 15 Nov 2022 12:16:11 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=CuhvpEh+MwcQ8DdyO2LHPKc75+Mh4Tt/SrTxun8SjsbCLbKA/uujVn75/hrSWRbq3P pp681Bo0ABWoVlpW4KZ12uslMC8XMyFowDj24K4rvmpM/9YujrW27P9Vv5MLxt+ZIfOV Jm6zqwFB/7ZHai0ByGJfyD247/5LpCxHhMpkaFu377RaNtmMs+gPLXFcf0G1yQbvs+Rs EYu+becUwCm5WaK54RGM6rYBIysavORuhi+fV9JxKIFpQITo1Jz1JNtjUgO7b5R2U1/e xEGMqJLbi5RxqShnMYvwl9BbYi//M2ZHV0GVHxnWgbMAYD9A/uq/G/wppECfL4cNv6ol hTbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=322wsvAp+Eo3teoA1Eak7iiE5Uj9pG6pl6oURDoj9RKANMX3qjbVkC1yWB2JJtEMIg cr0x7yXqrRoRvnneTIkqJXOIayrOuAFxYhsNYNDcQiMcvUz+3q0MmBtSuBk/YYcN3Y5I KosyrEeFBTGBSj0hzwC7wQIh2xYawD2vDsfduFzpC8meF591p9OIFn3R5brjHEIGPaHk IEF1OGJDgrtcwBiMVFeuCp+rqxWEGmQy8GyRNMCLrKFcKcJyJFYuCAefuK7Mq/tZGXKC 5QdI7piaZYuN5XRk6+EvihTvX2H/jcbLefKn3Q7TxI1wxgKQEuPgP3bhj3d7WYM+rBXC CIAA== X-Gm-Message-State: ANoB5pl5lrLIq2ofj5KA7iboKi5imkrbTRHova9TBykOAmydJd48a3pt IzdFicCEGglK8A3X9G1oEE82aQ== X-Google-Smtp-Source: AA0mqf58/qb3pHX8jXG9rQwsI0kWr5CgUXW3YysHviPdV0VVEN3OAELl4gQs2//98Y2csv6jhLKLCw== X-Received: by 2002:a17:902:e8d5:b0:181:6c64:6dd3 with SMTP id v21-20020a170902e8d500b001816c646dd3mr5671131plg.123.1668543368117; Tue, 15 Nov 2022 12:16:08 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id d62-20020a623641000000b0056c08c87196sm9173979pfa.48.2022.11.15.12.16.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:16:07 -0800 (PST) Date: Tue, 15 Nov 2022 20:16:04 +0000 From: Sean Christopherson To: "Huang, Kai" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "mjrosato@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "paul.walmsley@sifive.com" , "linux-mips@vger.kernel.org" , "linux-riscv@lists.infradead.org" , "imbrenda@linux.ibm.com" , "kvmarm@lists.cs.columbia.edu" , "linux-s390@vger.kernel.org" , "frankja@linux.ibm.com" , "maz@kernel.org" , "chenhuacai@kernel.org" , "aleksandar.qemu.devel@gmail.com" , "james.morse@arm.com" , "borntraeger@linux.ibm.com" , "Gao, Chao" , "farman@linux.ibm.com" , "aou@eecs.berkeley.edu" , "suzuki.poulose@arm.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "kvmarm@lists.linux.dev" , "tglx@linutronix.de" , "alexandru.elisei@arm.com" , "linux-arm-kernel@lists.infradead.org" , "Yamahata, Isaku" , "farosas@linux.ibm.com" , "linux-kernel@vger.kernel.org" , "oliver.upton@linux.dev" , "palmer@dabbelt.com" , "kvm-riscv@lists.infradead.org" , "anup@brainfault.org" , "pbonzini@redhat.com" , "vkuznets@redhat.com" , "linuxppc-dev@lists.ozlabs.org" Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > >   int cpu = smp_processor_id(); > > >   struct cpuinfo_x86 *c = &cpu_data(cpu); > > >   > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > >   > > > > Also, the logic of: > > > > !irqs_disabled() && cpu_active(cpu) > > > > is quite weird. > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > IRQ has already been enabled (see start_secondary()).  IIUC the WARN_ON() > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > check. > > > > So, a more reasonable check should be something like: > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > Or we can simply do: > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > ONLINE section). > > > > Btw above is purely based on code analysis, but I haven't done any test. > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > false. Actually, you're right (and wrong). You're right in that the WARN is flawed. And the reason for that is because you're wrong about the hotplug case. In this version of things, the compatibility checks are routed through hardware enabling, i.e. this flow is used only when loading KVM. This helper should only be called via SMP function call, which means that IRQs should always be disabled. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 7EA56C4332F for ; Tue, 15 Nov 2022 20:17:30 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=9saBvKFjuDEk8bF56rqdRrV3+C+V7ER69SQsWK/QfkU=; b=pMetufsKun+VAl DcTsPkJq85rVh1CNLfFAyZVtCNedCNW2lYTwFgGWXWZ1nvhJK5LGjsDs/3sK9CaebFyB9Sbf8o3yv bQWVufrPxPAU5qe780rW3KdsWmBprnhw6GzKyImKeXYqgJj6OEEbuXbQ0HgLJgpHE2HlnfUn2lsAK nYho+Kq3+uOFoFoOLc2VNDRKSl8RBHwOmzW6OPDay/sFtRRf7rcU3OBc9nOfQ3EvpIRpc3KQrxVwa JXmsgn2Ex0QA1+R4fxcdi67ybIL4iAbqoTT++0X/M7jLv1L98FITPO+6nVdiWVkodIxlMdpBaj2mW ZcJcfCmGrIM3QLE/uF2w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2Lw-00EVuo-Td; Tue, 15 Nov 2022 20:16:17 +0000 Received: from mail-pj1-x1032.google.com ([2607:f8b0:4864:20::1032]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2Lt-00EVsZ-AG for linux-arm-kernel@lists.infradead.org; Tue, 15 Nov 2022 20:16:15 +0000 Received: by mail-pj1-x1032.google.com with SMTP id b1-20020a17090a7ac100b00213fde52d49so14890629pjl.3 for ; Tue, 15 Nov 2022 12:16:08 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=CuhvpEh+MwcQ8DdyO2LHPKc75+Mh4Tt/SrTxun8SjsbCLbKA/uujVn75/hrSWRbq3P pp681Bo0ABWoVlpW4KZ12uslMC8XMyFowDj24K4rvmpM/9YujrW27P9Vv5MLxt+ZIfOV Jm6zqwFB/7ZHai0ByGJfyD247/5LpCxHhMpkaFu377RaNtmMs+gPLXFcf0G1yQbvs+Rs EYu+becUwCm5WaK54RGM6rYBIysavORuhi+fV9JxKIFpQITo1Jz1JNtjUgO7b5R2U1/e xEGMqJLbi5RxqShnMYvwl9BbYi//M2ZHV0GVHxnWgbMAYD9A/uq/G/wppECfL4cNv6ol hTbw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=T7LP3RGpmdXiMN5KPTn9/IO+rih2AglOTsvbEdRvS1Q=; b=BXVeZl9sMvV/VrOTxQhNte/asWPG503IM6sKHTPP5ih45xyy1css15d/Q+4VyFFFpv ZAcYxHfwzSn+pdtmL8/PHaTbE5NZDFJiTq26ytvaz4/NBSj4TtfyNWphaCZ1rLYwh5A0 yzQqsUxh6VHqSbYuSiwsG8FOUjNcIEGUqWhdhu9SmxP/U7W7lqAbacUju4lkoS1j+SC9 7onghRMFShNhqL71Enx2XUdTA6PyIkhRGQecur5E1xQDuvTLCWbvcB9NpX7l79pMAH9M B/Xvziq3o6hJ3CZu6AMhGdw3NCOhsIAYb6oQjRu50faEfNbAbjmm82SlAp9O4nNqs4uR dDKg== X-Gm-Message-State: ANoB5pnsOd0howM7nnGplYPSn8pSheZECL9Vyz0Bh7+0c2gsaYVmEDHQ nG1MMD3ZnlJF156ifMvWYQq5iA== X-Google-Smtp-Source: AA0mqf58/qb3pHX8jXG9rQwsI0kWr5CgUXW3YysHviPdV0VVEN3OAELl4gQs2//98Y2csv6jhLKLCw== X-Received: by 2002:a17:902:e8d5:b0:181:6c64:6dd3 with SMTP id v21-20020a170902e8d500b001816c646dd3mr5671131plg.123.1668543368117; Tue, 15 Nov 2022 12:16:08 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id d62-20020a623641000000b0056c08c87196sm9173979pfa.48.2022.11.15.12.16.07 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:16:07 -0800 (PST) Date: Tue, 15 Nov 2022 20:16:04 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221115_121613_378124_4FD2A3A9 X-CRM114-Status: GOOD ( 25.41 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, Nov 10, 2022, Huang, Kai wrote: > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > @@ -9283,7 +9283,13 @@ static int > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > =A0 int cpu =3D smp_processor_id(); > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > =A0 > > > - WARN_ON(!irqs_disabled()); > > > + /* > > > + * Compatibility checks are done when loading KVM and when enabling > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > + * compatible, i.e. KVM should never perform a compatibility check > > > on > > > + * an offline CPU. > > > + */ > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > =A0 > > = > > Also, the logic of: > > = > > !irqs_disabled() && cpu_active(cpu) > > = > > is quite weird. > > = > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > section > > the IRQ is indeed disabled. > > = > > But this doesn't make sense anymore after we move to ONLINE section, in= which > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WARN_= ON() > > doesn't get exploded is purely because there's an additional cpu_active= (cpu) > > check. > > = > > So, a more reasonable check should be something like: > > = > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > = > > Or we can simply do: > > = > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > = > > (because I don't know whether it's possible IRQ can somehow get disable= d in > > ONLINE section). > > = > > Btw above is purely based on code analysis, but I haven't done any test. > = > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > happens on all online cpus when loading KVM. For this case, IRQ is disab= led and > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_acti= ve() is > false. Actually, you're right (and wrong). You're right in that the WARN is flawe= d. And the reason for that is because you're wrong about the hotplug case. In thi= s version of things, the compatibility checks are routed through hardware enabling, i= .e. this flow is used only when loading KVM. This helper should only be called via = SMP function call, which means that IRQs should always be disabled. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel