From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Tue, 15 Nov 2022 20:21:00 +0000 Subject: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling In-Reply-To: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Tue, Nov 15, 2022, Sean Christopherson wrote: > On Thu, Nov 10, 2022, Huang, Kai wrote: > > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > > @@ -9283,7 +9283,13 @@ static int > > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > > ? int cpu = smp_processor_id(); > > > > ? struct cpuinfo_x86 *c = &cpu_data(cpu); > > > > ? > > > > - WARN_ON(!irqs_disabled()); > > > > + /* > > > > + * Compatibility checks are done when loading KVM and when enabling > > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > > + * compatible, i.e. KVM should never perform a compatibility check > > > > on > > > > + * an offline CPU. > > > > + */ > > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > > ? > > > > > > Also, the logic of: > > > > > > !irqs_disabled() && cpu_active(cpu) > > > > > > is quite weird. > > > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > > section > > > the IRQ is indeed disabled. > > > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > > IRQ has already been enabled (see start_secondary()).? IIUC the WARN_ON() > > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > > check. > > > > > > So, a more reasonable check should be something like: > > > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > > > Or we can simply do: > > > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > > ONLINE section). > > > > > > Btw above is purely based on code analysis, but I haven't done any test. > > > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > > false. > > Actually, you're right (and wrong). You're right in that the WARN is flawed. And > the reason for that is because you're wrong about the hotplug case. In this version > of things, the compatibility checks are routed through hardware enabling, i.e. this > flow is used only when loading KVM. This helper should only be called via SMP function > call, which means that IRQs should always be disabled. Grr, but not routing through this helper is flawed in that KVM doesn't do the CR4 checks in the hardware enabling case. Don't think that changes the WARN, but other patches in this series need tweaks. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id D38B4C4332F for ; Tue, 15 Nov 2022 20:21:09 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 32CFD4B8D9; Tue, 15 Nov 2022 15:21:09 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id AfmV9pT-us-M; Tue, 15 Nov 2022 15:21:08 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 1E0174B8C6; Tue, 15 Nov 2022 15:21:08 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 750484B8B8 for ; Tue, 15 Nov 2022 15:21:06 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id ES23f3cQGwuK for ; Tue, 15 Nov 2022 15:21:05 -0500 (EST) Received: from mail-pg1-f175.google.com (mail-pg1-f175.google.com [209.85.215.175]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id 640B04B890 for ; Tue, 15 Nov 2022 15:21:05 -0500 (EST) Received: by mail-pg1-f175.google.com with SMTP id 62so6025050pgb.13 for ; Tue, 15 Nov 2022 12:21:05 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=QoLsrppxCqrrm8Et/yxXiG7qWE0hiOrc+RLWkwKVvGYvEXTURW54Ct7G0aFnW7vDom O3WnXZWOxSKhgS+uv00LIKaFYAiY4w1O0HuVw0sHlWnVNkXyDKqsQfpccjPy0o//76Ot 9HbC7sv+GYjRBhUyAvWsaFUrflZFb6BoKafnLIcIXy0vGyf6tKbOGR6vv68o6WUCslEQ BOT4pCgeSJnKExa2xfKvrZGwBZhzuokrCi6o4yMRfhSIGNbmSd/1S94WiYpeZI3RH1jS tMxrZXr8PoWD2Hx+ZMuaUIAmz2hZjOEiOFHGi/ELuekxNrIkmFIAeW4IyLdTJL7wajgf 6FdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=H6QMtgwi++hKy4NyIh/cw8whQJoA7wsW0DmmYbI3Qwo6Pm2ImYjswEriFVSBUbN/uK 4i7iXClsRRdsakEL3fFdWILoyszb+nxqJK/8uCFHD1ol5QX5oWIk7LtZLFhc1e3d5Fyv 3+Gu6TWPbp5ZNbU36R2Uv5EAdwFkVlL0byhbbhBQhNVzOSruUJQe0/Ccwxx7EotY6Tch dMGGAzEP1i+enKrLZU6Q7oC0ypYv0dSwKZbVyDuax0AelsVBx+Vl8IVCFPLjxRMSy6aQ xM0RaO7YOFoyu4hCB6Dq2wnHtDS0Pv7gg5vW6YWgAcYmD2R2uAHtAy305ZCbHjLdCna9 gVBw== X-Gm-Message-State: ANoB5plZRetnAlAeb4Y6iDBfjhGiDellFjxe2MOKTfW3gROe2pOtaK7T ELapzyLOcILkuqkoQnLucotNiQ== X-Google-Smtp-Source: AA0mqf7I+HCT8JLUK2SnELicaIfj+5Xh/wdDfnKOa9DuiFN7rhFS7bzZLz9XVMfRQzvW4tmCjZQXQw== X-Received: by 2002:aa7:9493:0:b0:56b:9ae8:ca05 with SMTP id z19-20020aa79493000000b0056b9ae8ca05mr19812088pfk.59.1668543664233; Tue, 15 Nov 2022 12:21:04 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b001754fa42065sm10448477plg.143.2022.11.15.12.21.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:21:03 -0800 (PST) Date: Tue, 15 Nov 2022 20:21:00 +0000 From: Sean Christopherson To: "Huang, Kai" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: Cc: "mjrosato@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "paul.walmsley@sifive.com" , "linux-mips@vger.kernel.org" , "mpe@ellerman.id.au" , "linux-riscv@lists.infradead.org" , "imbrenda@linux.ibm.com" , "kvmarm@lists.cs.columbia.edu" , "linux-s390@vger.kernel.org" , "frankja@linux.ibm.com" , "maz@kernel.org" , "chenhuacai@kernel.org" , "aleksandar.qemu.devel@gmail.com" , "borntraeger@linux.ibm.com" , "Gao, Chao" , "farman@linux.ibm.com" , "aou@eecs.berkeley.edu" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "kvmarm@lists.linux.dev" , "tglx@linutronix.de" , "linux-arm-kernel@lists.infradead.org" , "Yamahata, Isaku" , "farosas@linux.ibm.com" , "linux-kernel@vger.kernel.org" , "palmer@dabbelt.com" , "kvm-riscv@lists.infradead.org" , "pbonzini@redhat.com" , "vkuznets@redhat.com" , "linuxppc-dev@lists.ozlabs.org" X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Tue, Nov 15, 2022, Sean Christopherson wrote: > On Thu, Nov 10, 2022, Huang, Kai wrote: > > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > > @@ -9283,7 +9283,13 @@ static int > > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > > =A0 int cpu =3D smp_processor_id(); > > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > > =A0 > > > > - WARN_ON(!irqs_disabled()); > > > > + /* > > > > + * Compatibility checks are done when loading KVM and when enabli= ng > > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs a= re > > > > + * compatible, i.e. KVM should never perform a compatibility check > > > > on > > > > + * an offline CPU. > > > > + */ > > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > > =A0 > > > = > > > Also, the logic of: > > > = > > > !irqs_disabled() && cpu_active(cpu) > > > = > > > is quite weird. > > > = > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTI= NG > > > section > > > the IRQ is indeed disabled. > > > = > > > But this doesn't make sense anymore after we move to ONLINE section, = in which > > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WAR= N_ON() > > > doesn't get exploded is purely because there's an additional cpu_acti= ve(cpu) > > > check. > > > = > > > So, a more reasonable check should be something like: > > > = > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > = > > > Or we can simply do: > > > = > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > = > > > (because I don't know whether it's possible IRQ can somehow get disab= led in > > > ONLINE section). > > > = > > > Btw above is purely based on code analysis, but I haven't done any te= st. > > = > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check a= lso > > happens on all online cpus when loading KVM. For this case, IRQ is dis= abled and > > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_ac= tive() is > > false. > = > Actually, you're right (and wrong). You're right in that the WARN is fla= wed. And > the reason for that is because you're wrong about the hotplug case. In t= his version > of things, the compatibility checks are routed through hardware enabling,= i.e. this > flow is used only when loading KVM. This helper should only be called vi= a SMP function > call, which means that IRQs should always be disabled. Grr, but not routing through this helper is flawed in that KVM doesn't do t= he CR4 checks in the hardware enabling case. Don't think that changes the WAR= N, but other patches in this series need tweaks. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from mail-pg1-f180.google.com (mail-pg1-f180.google.com [209.85.215.180]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by smtp.subspace.kernel.org (Postfix) with ESMTPS id 123BE8ABA for ; Tue, 15 Nov 2022 20:21:04 +0000 (UTC) Received: by mail-pg1-f180.google.com with SMTP id n17so6734716pgh.9 for ; Tue, 15 Nov 2022 12:21:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=QoLsrppxCqrrm8Et/yxXiG7qWE0hiOrc+RLWkwKVvGYvEXTURW54Ct7G0aFnW7vDom O3WnXZWOxSKhgS+uv00LIKaFYAiY4w1O0HuVw0sHlWnVNkXyDKqsQfpccjPy0o//76Ot 9HbC7sv+GYjRBhUyAvWsaFUrflZFb6BoKafnLIcIXy0vGyf6tKbOGR6vv68o6WUCslEQ BOT4pCgeSJnKExa2xfKvrZGwBZhzuokrCi6o4yMRfhSIGNbmSd/1S94WiYpeZI3RH1jS tMxrZXr8PoWD2Hx+ZMuaUIAmz2hZjOEiOFHGi/ELuekxNrIkmFIAeW4IyLdTJL7wajgf 6FdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=xEorbhymHS6Aos4evNVnoYgcLa5XXxLlQhtZDjRsJWcQAlYfLGc8qqVl10xD3ENfc7 BHm85UCgQab1+QMlX14HJ9H/Xz/eQaHr3PRt4fpqSDbDIWGg5xx3bOLpimCQEEJgA7GH Mjtt/r1wggTHbhUftsFKKAC6dtzDpx1iybmLSvB87jN8MUqwgrOacTpsiHLUkPsvAoku LQPbjT+nH8YlJNhhxPcAA0vE/Xt9pTcdzwn52vblLv7vtbQot5p0SchgwFc1OnfrmaSw Lj35QvHq7lr1SYPxW0TpP4SoANpa9FGOKoCgPXkrC8kYx/UgPBOG4xWd6f6uS9DhFCO3 5D/A== X-Gm-Message-State: ANoB5pnec4iCfrA/sM1rj/9sUv2487eGrlSm6rh7CIGiuqDVEfQh6r8w kq5JdBkBanbZSbzgK5eR14RHqQ== X-Google-Smtp-Source: AA0mqf7I+HCT8JLUK2SnELicaIfj+5Xh/wdDfnKOa9DuiFN7rhFS7bzZLz9XVMfRQzvW4tmCjZQXQw== X-Received: by 2002:aa7:9493:0:b0:56b:9ae8:ca05 with SMTP id z19-20020aa79493000000b0056b9ae8ca05mr19812088pfk.59.1668543664233; Tue, 15 Nov 2022 12:21:04 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b001754fa42065sm10448477plg.143.2022.11.15.12.21.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:21:03 -0800 (PST) Date: Tue, 15 Nov 2022 20:21:00 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> Precedence: bulk X-Mailing-List: kvmarm@lists.linux.dev List-Id: List-Subscribe: List-Unsubscribe: MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: Message-ID: <20221115202100.4nbK41kraAo0ZKuxO9EklTl2rZpLVoVJuUuTJ-KOPKY@z> On Tue, Nov 15, 2022, Sean Christopherson wrote: > On Thu, Nov 10, 2022, Huang, Kai wrote: > > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > > @@ -9283,7 +9283,13 @@ static int > > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > >   int cpu = smp_processor_id(); > > > >   struct cpuinfo_x86 *c = &cpu_data(cpu); > > > >   > > > > - WARN_ON(!irqs_disabled()); > > > > + /* > > > > + * Compatibility checks are done when loading KVM and when enabling > > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > > + * compatible, i.e. KVM should never perform a compatibility check > > > > on > > > > + * an offline CPU. > > > > + */ > > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > >   > > > > > > Also, the logic of: > > > > > > !irqs_disabled() && cpu_active(cpu) > > > > > > is quite weird. > > > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > > section > > > the IRQ is indeed disabled. > > > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > > IRQ has already been enabled (see start_secondary()).  IIUC the WARN_ON() > > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > > check. > > > > > > So, a more reasonable check should be something like: > > > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > > > Or we can simply do: > > > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > > ONLINE section). > > > > > > Btw above is purely based on code analysis, but I haven't done any test. > > > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > > false. > > Actually, you're right (and wrong). You're right in that the WARN is flawed. And > the reason for that is because you're wrong about the hotplug case. In this version > of things, the compatibility checks are routed through hardware enabling, i.e. this > flow is used only when loading KVM. This helper should only be called via SMP function > call, which means that IRQs should always be disabled. Grr, but not routing through this helper is flawed in that KVM doesn't do the CR4 checks in the hardware enabling case. Don't think that changes the WARN, but other patches in this series need tweaks. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 52C96C433FE for ; Tue, 15 Nov 2022 20:21:17 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=IR996dpAAabwYqzxgMLOVZ1MoAqbtvi68dXCAuP4I+I=; b=X/rGn5RWtq/qjt RRjl2nnfe8TWSzrP0ARXVWFn16nW04SPE8AtoEmAURCiHT3VHPCMQI3XaculOeH3vrBylH1JjgSeB bN179nz1K/UKnNWQXMn09zs9AvJBMHMTYPrx97kYvCBFWlk0N9qh0tHAMCYcjwHrt7zvjotosGVmk /SBZj5JtSeSxVd57kmqT/4A9R7ISl04yJZT64oJgbOxf4X/re1WN1nZd8JpSm36jn/oeeTTe7eTfa 1JgA7bHeXClSs9jwY8fZHugRE/YiHhBvzR318sZLcdHTrckxTR4xpRMFb5bkV6FZgsD3yPO/PhlY8 oCvZKvzouCMd74Ix5kjA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2Qg-00EX6P-3W; Tue, 15 Nov 2022 20:21:10 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2Qc-00EX4k-1P for linux-riscv@lists.infradead.org; Tue, 15 Nov 2022 20:21:08 +0000 Received: by mail-pf1-x429.google.com with SMTP id c203so4003552pfc.11 for ; Tue, 15 Nov 2022 12:21:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=QoLsrppxCqrrm8Et/yxXiG7qWE0hiOrc+RLWkwKVvGYvEXTURW54Ct7G0aFnW7vDom O3WnXZWOxSKhgS+uv00LIKaFYAiY4w1O0HuVw0sHlWnVNkXyDKqsQfpccjPy0o//76Ot 9HbC7sv+GYjRBhUyAvWsaFUrflZFb6BoKafnLIcIXy0vGyf6tKbOGR6vv68o6WUCslEQ BOT4pCgeSJnKExa2xfKvrZGwBZhzuokrCi6o4yMRfhSIGNbmSd/1S94WiYpeZI3RH1jS tMxrZXr8PoWD2Hx+ZMuaUIAmz2hZjOEiOFHGi/ELuekxNrIkmFIAeW4IyLdTJL7wajgf 6FdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=L7auKaW5dON9Z6pvzfucApIvcbqIttyVrTdH3eBRbWBbg8Ez6XepFnX9XxBSNHrHfF esvlECzcL4VeqpdYn26IKnqTmZLdCr4Q8LDi8BObfJt7sO/eLlZS2DEPER5SwvawpC24 cPWvbPvBDfrlrsPUnu3URjr//b9zU5DY0jWoqbNv6dPf8P1t8k0e4ifRgQPVf7fY/N5q KByIxdHURub2TYtjzET205R9q1OMsymghnESH9YUZkATWqVnPL70621cfab5MSB1xHfG 8lLwn5qfmEG1GJiBh5ZTglqHvR0NVuz2bEVi+nXxkzbLhPQiwK1YyLjMBcAJLjGOy+hJ /nfg== X-Gm-Message-State: ANoB5pljO9IE0Vm3jtYvQUBRbplB5jwgNUDQI75i+B3RVcP1Ssv5qfRl 7yA3jSPS7NJatUv02JBKT4pVVQ== X-Google-Smtp-Source: AA0mqf7I+HCT8JLUK2SnELicaIfj+5Xh/wdDfnKOa9DuiFN7rhFS7bzZLz9XVMfRQzvW4tmCjZQXQw== X-Received: by 2002:aa7:9493:0:b0:56b:9ae8:ca05 with SMTP id z19-20020aa79493000000b0056b9ae8ca05mr19812088pfk.59.1668543664233; Tue, 15 Nov 2022 12:21:04 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b001754fa42065sm10448477plg.143.2022.11.15.12.21.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:21:03 -0800 (PST) Date: Tue, 15 Nov 2022 20:21:00 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221115_122106_108375_52BB820A X-CRM114-Status: GOOD ( 27.64 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Tue, Nov 15, 2022, Sean Christopherson wrote: > On Thu, Nov 10, 2022, Huang, Kai wrote: > > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > > @@ -9283,7 +9283,13 @@ static int > > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > > =A0 int cpu =3D smp_processor_id(); > > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > > =A0 > > > > - WARN_ON(!irqs_disabled()); > > > > + /* > > > > + * Compatibility checks are done when loading KVM and when enabli= ng > > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs a= re > > > > + * compatible, i.e. KVM should never perform a compatibility check > > > > on > > > > + * an offline CPU. > > > > + */ > > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > > =A0 > > > = > > > Also, the logic of: > > > = > > > !irqs_disabled() && cpu_active(cpu) > > > = > > > is quite weird. > > > = > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTI= NG > > > section > > > the IRQ is indeed disabled. > > > = > > > But this doesn't make sense anymore after we move to ONLINE section, = in which > > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WAR= N_ON() > > > doesn't get exploded is purely because there's an additional cpu_acti= ve(cpu) > > > check. > > > = > > > So, a more reasonable check should be something like: > > > = > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > = > > > Or we can simply do: > > > = > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > = > > > (because I don't know whether it's possible IRQ can somehow get disab= led in > > > ONLINE section). > > > = > > > Btw above is purely based on code analysis, but I haven't done any te= st. > > = > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check a= lso > > happens on all online cpus when loading KVM. For this case, IRQ is dis= abled and > > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_ac= tive() is > > false. > = > Actually, you're right (and wrong). You're right in that the WARN is fla= wed. And > the reason for that is because you're wrong about the hotplug case. In t= his version > of things, the compatibility checks are routed through hardware enabling,= i.e. this > flow is used only when loading KVM. This helper should only be called vi= a SMP function > call, which means that IRQs should always be disabled. Grr, but not routing through this helper is flawed in that KVM doesn't do t= he CR4 checks in the hardware enabling case. Don't think that changes the WAR= N, but other patches in this series need tweaks. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from lists.ozlabs.org (lists.ozlabs.org [112.213.38.117]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E3EC5C4332F for ; Tue, 15 Nov 2022 20:22:03 +0000 (UTC) Received: from boromir.ozlabs.org (localhost [IPv6:::1]) by lists.ozlabs.org (Postfix) with ESMTP id 4NBczG0Gl6z3cN0 for ; Wed, 16 Nov 2022 07:22:02 +1100 (AEDT) Authentication-Results: lists.ozlabs.org; dkim=fail reason="signature verification failed" (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=QoLsrppx; dkim-atps=neutral Authentication-Results: lists.ozlabs.org; spf=pass (sender SPF authorized) smtp.mailfrom=google.com (client-ip=2607:f8b0:4864:20::52b; helo=mail-pg1-x52b.google.com; envelope-from=seanjc@google.com; receiver=) Authentication-Results: lists.ozlabs.org; dkim=pass (2048-bit key; unprotected) header.d=google.com header.i=@google.com header.a=rsa-sha256 header.s=20210112 header.b=QoLsrppx; dkim-atps=neutral Received: from mail-pg1-x52b.google.com (mail-pg1-x52b.google.com [IPv6:2607:f8b0:4864:20::52b]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 4NBcyC54Kvz3bXR for ; Wed, 16 Nov 2022 07:21:07 +1100 (AEDT) Received: by mail-pg1-x52b.google.com with SMTP id b62so14403010pgc.0 for ; Tue, 15 Nov 2022 12:21:07 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=QoLsrppxCqrrm8Et/yxXiG7qWE0hiOrc+RLWkwKVvGYvEXTURW54Ct7G0aFnW7vDom O3WnXZWOxSKhgS+uv00LIKaFYAiY4w1O0HuVw0sHlWnVNkXyDKqsQfpccjPy0o//76Ot 9HbC7sv+GYjRBhUyAvWsaFUrflZFb6BoKafnLIcIXy0vGyf6tKbOGR6vv68o6WUCslEQ BOT4pCgeSJnKExa2xfKvrZGwBZhzuokrCi6o4yMRfhSIGNbmSd/1S94WiYpeZI3RH1jS tMxrZXr8PoWD2Hx+ZMuaUIAmz2hZjOEiOFHGi/ELuekxNrIkmFIAeW4IyLdTJL7wajgf 6FdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=4DcYHe8ATAV2sZzPBSQ7Euh8ngxrQpbK/Lul7nXDxGC2z3jC7rvYUNblgfYx8+Pf7/ DAQEQ5dtwNxku7i9e1HD/azn0HagMJ4MiEl2CM0kuaad03gkMNZr4/NTuzLp036sXg+M YftvHc0dMZFtMJFq9zK/PP/X9Gcj+RcbCF0/+XqCbcQ1AI0jPKuTm5WJIvNlwHfGnp8U NsccfewuBrtQwEvA/BI+qMO+ke7njzV6tDTT5MhVbe00K8VNJvrx8Kx3h54T7zWATFH/ ItX3F6by6/NOlrfjwLcrSnIfx0LPlLvzwpnfCKziHjNJMMtuNc7H7+00zoOuZxaVzFv5 KKDw== X-Gm-Message-State: ANoB5pnISWLBqFWQVLmAkCGcwFTU43hYuO/zilbo56cE4hXsHH4fPG3Y Og3fTm5E9iBjE6LJ0NSmxYArMg== X-Google-Smtp-Source: AA0mqf7I+HCT8JLUK2SnELicaIfj+5Xh/wdDfnKOa9DuiFN7rhFS7bzZLz9XVMfRQzvW4tmCjZQXQw== X-Received: by 2002:aa7:9493:0:b0:56b:9ae8:ca05 with SMTP id z19-20020aa79493000000b0056b9ae8ca05mr19812088pfk.59.1668543664233; Tue, 15 Nov 2022 12:21:04 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b001754fa42065sm10448477plg.143.2022.11.15.12.21.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:21:03 -0800 (PST) Date: Tue, 15 Nov 2022 20:21:00 +0000 From: Sean Christopherson To: "Huang, Kai" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-BeenThere: linuxppc-dev@lists.ozlabs.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "mjrosato@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "paul.walmsley@sifive.com" , "linux-mips@vger.kernel.org" , "linux-riscv@lists.infradead.org" , "imbrenda@linux.ibm.com" , "kvmarm@lists.cs.columbia.edu" , "linux-s390@vger.kernel.org" , "frankja@linux.ibm.com" , "maz@kernel.org" , "chenhuacai@kernel.org" , "aleksandar.qemu.devel@gmail.com" , "james.morse@arm.com" , "borntraeger@linux.ibm.com" , "Gao, Chao" , "farman@linux.ibm.com" , "aou@eecs.berkeley.edu" , "suzuki.poulose@arm.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "kvmarm@lists.linux.dev" , "tglx@linutronix.de" , "alexandru.elisei@arm.com" , "linux-arm-kernel@lists.infradead.org" , "Yamahata, Isaku" , "farosas@linux.ibm.com" , "linux-kernel@vger.kernel.org" , "oliver.upton@linux.dev" , "palmer@dabbelt.com" , "kvm-riscv@lists.infradead.org" , "anup@brainfault.org" , "pbonzini@redhat.com" , "vkuznets@redhat.com" , "linuxppc-dev@lists.ozlabs.org" Errors-To: linuxppc-dev-bounces+linuxppc-dev=archiver.kernel.org@lists.ozlabs.org Sender: "Linuxppc-dev" On Tue, Nov 15, 2022, Sean Christopherson wrote: > On Thu, Nov 10, 2022, Huang, Kai wrote: > > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > > @@ -9283,7 +9283,13 @@ static int > > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > >   int cpu = smp_processor_id(); > > > >   struct cpuinfo_x86 *c = &cpu_data(cpu); > > > >   > > > > - WARN_ON(!irqs_disabled()); > > > > + /* > > > > + * Compatibility checks are done when loading KVM and when enabling > > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs are > > > > + * compatible, i.e. KVM should never perform a compatibility check > > > > on > > > > + * an offline CPU. > > > > + */ > > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > >   > > > > > > Also, the logic of: > > > > > > !irqs_disabled() && cpu_active(cpu) > > > > > > is quite weird. > > > > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTING > > > section > > > the IRQ is indeed disabled. > > > > > > But this doesn't make sense anymore after we move to ONLINE section, in which > > > IRQ has already been enabled (see start_secondary()).  IIUC the WARN_ON() > > > doesn't get exploded is purely because there's an additional cpu_active(cpu) > > > check. > > > > > > So, a more reasonable check should be something like: > > > > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > > > > Or we can simply do: > > > > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > > > > (because I don't know whether it's possible IRQ can somehow get disabled in > > > ONLINE section). > > > > > > Btw above is purely based on code analysis, but I haven't done any test. > > > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check also > > happens on all online cpus when loading KVM. For this case, IRQ is disabled and > > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_active() is > > false. > > Actually, you're right (and wrong). You're right in that the WARN is flawed. And > the reason for that is because you're wrong about the hotplug case. In this version > of things, the compatibility checks are routed through hardware enabling, i.e. this > flow is used only when loading KVM. This helper should only be called via SMP function > call, which means that IRQs should always be disabled. Grr, but not routing through this helper is flawed in that KVM doesn't do the CR4 checks in the hardware enabling case. Don't think that changes the WARN, but other patches in this series need tweaks. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B9793C433FE for ; Tue, 15 Nov 2022 20:22:23 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=ukGJoWe4K6ekqGwY4xOG4Oim97ZV8IXBnBQ6rxsCOHk=; b=E4r+kPKtwZpqyh 4YppSVFBrd7lwUy3IeZPBY7Vb7s8YO+W/vKq6ibmQ7XUkXRotpQJ/Wk/JAC9Ycmi15/5qvgw0OAzd 3fRIpe30Vvw3h0KpvGmYtTOWSWmFDPPC6Yx1aiHdCJmsqAIeighC2sGELftfc287N2Q3MEX+zFbxD OhRyMuVUveJdQZbzCLFV8/ndWp+/Fb6+/8AmaWOi6tj8LL3LnKC29FmoJEuHSt0vQN95TtWIOas12 +lB+QS4G4Ad9iI9fLCsWu6GmRyGK0oE3Ho9HpaPJL0oteusyefmr0+o6bntMaEr30I4OXa3yEVp5i hz/VvTGmkRkrx4kRm2iA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2Qi-00EX7Y-5l; Tue, 15 Nov 2022 20:21:12 +0000 Received: from mail-pf1-x429.google.com ([2607:f8b0:4864:20::429]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ov2Qc-00EX4j-1Q for linux-arm-kernel@lists.infradead.org; Tue, 15 Nov 2022 20:21:08 +0000 Received: by mail-pf1-x429.google.com with SMTP id b29so15100734pfp.13 for ; Tue, 15 Nov 2022 12:21:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=QoLsrppxCqrrm8Et/yxXiG7qWE0hiOrc+RLWkwKVvGYvEXTURW54Ct7G0aFnW7vDom O3WnXZWOxSKhgS+uv00LIKaFYAiY4w1O0HuVw0sHlWnVNkXyDKqsQfpccjPy0o//76Ot 9HbC7sv+GYjRBhUyAvWsaFUrflZFb6BoKafnLIcIXy0vGyf6tKbOGR6vv68o6WUCslEQ BOT4pCgeSJnKExa2xfKvrZGwBZhzuokrCi6o4yMRfhSIGNbmSd/1S94WiYpeZI3RH1jS tMxrZXr8PoWD2Hx+ZMuaUIAmz2hZjOEiOFHGi/ELuekxNrIkmFIAeW4IyLdTJL7wajgf 6FdQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=XxrH205VSL6PuZxNZy5QSHBg1esXqApruwRLQHoD1FQ=; b=3givj7uHvKy4jYOsOcMHzk1p4G8wXPM0x8uGlxiRjPsRHEywTohd9aZ8gVECs1PkZE CMLgCMFIrtrzz5LiTkb6gy5aYHRkbIBn0zOlaDr2ct6tE2zPuTuRG/njM7OXZJwu18YH 8g8KiU5AuXUVHQM5GvaB7seRyFIroje6nM81pAzXvp+Tls+7vrmNguUfpVGVJoo04T1k wWiF7vg7U9P95F7Xf/l92hu+fCnsZUpKY5JfywXMGih2nXw8fkABVTKHaZaAtQQEYcsd znjCgpeh6DpGkafgNsln51VQvIOTuxzkk2PyhCuEIpCfJfcyzW44a6277XEH8cz8GWPi zjaQ== X-Gm-Message-State: ANoB5plpLIakjCzMfnFrYdNy9bTc+x/eSTOx2wAAJzAsobUF39ElZ/go KeO66VGa8IxQ0aFK/7rSJHyqWQ== X-Google-Smtp-Source: AA0mqf7I+HCT8JLUK2SnELicaIfj+5Xh/wdDfnKOa9DuiFN7rhFS7bzZLz9XVMfRQzvW4tmCjZQXQw== X-Received: by 2002:aa7:9493:0:b0:56b:9ae8:ca05 with SMTP id z19-20020aa79493000000b0056b9ae8ca05mr19812088pfk.59.1668543664233; Tue, 15 Nov 2022 12:21:04 -0800 (PST) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id o15-20020a170902d4cf00b001754fa42065sm10448477plg.143.2022.11.15.12.21.03 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 15 Nov 2022 12:21:03 -0800 (PST) Date: Tue, 15 Nov 2022 20:21:00 +0000 From: Sean Christopherson To: "Huang, Kai" Cc: "farman@linux.ibm.com" , "frankja@linux.ibm.com" , "mjrosato@linux.ibm.com" , "vkuznets@redhat.com" , "chenhuacai@kernel.org" , "aou@eecs.berkeley.edu" , "palmer@dabbelt.com" , "paul.walmsley@sifive.com" , "maz@kernel.org" , "anup@brainfault.org" , "imbrenda@linux.ibm.com" , "pbonzini@redhat.com" , "borntraeger@linux.ibm.com" , "aleksandar.qemu.devel@gmail.com" , "kvm@vger.kernel.org" , "atishp@atishpatra.org" , "farosas@linux.ibm.com" , "david@redhat.com" , "Yao, Yuan" , "mpe@ellerman.id.au" , "alexandru.elisei@arm.com" , "linux-s390@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "tglx@linutronix.de" , "Yamahata, Isaku" , "kvmarm@lists.linux.dev" , "james.morse@arm.com" , "kvm-riscv@lists.infradead.org" , "suzuki.poulose@arm.com" , "linuxppc-dev@lists.ozlabs.org" , "linux-arm-kernel@lists.infradead.org" , "linux-mips@vger.kernel.org" , "Gao, Chao" , "oliver.upton@linux.dev" , "kvmarm@lists.cs.columbia.edu" , "linux-riscv@lists.infradead.org" Subject: Re: [PATCH 38/44] KVM: Disable CPU hotplug during hardware enabling Message-ID: References: <20221102231911.3107438-1-seanjc@google.com> <20221102231911.3107438-39-seanjc@google.com> <88e920944de70e7d69a98f74005b49c59b5aaa3b.camel@intel.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20221115_122106_108458_5F27F6FE X-CRM114-Status: GOOD ( 29.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Tue, Nov 15, 2022, Sean Christopherson wrote: > On Thu, Nov 10, 2022, Huang, Kai wrote: > > On Thu, 2022-11-10 at 01:33 +0000, Huang, Kai wrote: > > > > @@ -9283,7 +9283,13 @@ static int > > > > kvm_x86_check_processor_compatibility(struct kvm_x86_init_ops *ops) > > > > =A0 int cpu =3D smp_processor_id(); > > > > =A0 struct cpuinfo_x86 *c =3D &cpu_data(cpu); > > > > =A0 > > > > - WARN_ON(!irqs_disabled()); > > > > + /* > > > > + * Compatibility checks are done when loading KVM and when enabli= ng > > > > + * hardware, e.g. during CPU hotplug, to ensure all online CPUs a= re > > > > + * compatible, i.e. KVM should never perform a compatibility check > > > > on > > > > + * an offline CPU. > > > > + */ > > > > + WARN_ON(!irqs_disabled() && cpu_active(cpu)); > > > > =A0 > > > = > > > Also, the logic of: > > > = > > > !irqs_disabled() && cpu_active(cpu) > > > = > > > is quite weird. > > > = > > > The original "WARN(!irqs_disabled())" is reasonable because in STARTI= NG > > > section > > > the IRQ is indeed disabled. > > > = > > > But this doesn't make sense anymore after we move to ONLINE section, = in which > > > IRQ has already been enabled (see start_secondary()).=A0 IIUC the WAR= N_ON() > > > doesn't get exploded is purely because there's an additional cpu_acti= ve(cpu) > > > check. > > > = > > > So, a more reasonable check should be something like: > > > = > > > WARN_ON(irqs_disabled() || cpu_active(cpu) || !cpu_online(cpu)); > > > = > > > Or we can simply do: > > > = > > > WARN_ON(!cpu_online(cpu) || cpu_active(cpu)); > > > = > > > (because I don't know whether it's possible IRQ can somehow get disab= led in > > > ONLINE section). > > > = > > > Btw above is purely based on code analysis, but I haven't done any te= st. > > = > > Hmm.. I wasn't thinking thoroughly. I forgot CPU compatibility check a= lso > > happens on all online cpus when loading KVM. For this case, IRQ is dis= abled and > > cpu_active() is true. For the hotplug case, IRQ is enabled but cpu_ac= tive() is > > false. > = > Actually, you're right (and wrong). You're right in that the WARN is fla= wed. And > the reason for that is because you're wrong about the hotplug case. In t= his version > of things, the compatibility checks are routed through hardware enabling,= i.e. this > flow is used only when loading KVM. This helper should only be called vi= a SMP function > call, which means that IRQs should always be disabled. Grr, but not routing through this helper is flawed in that KVM doesn't do t= he CR4 checks in the hardware enabling case. Don't think that changes the WAR= N, but other patches in this series need tweaks. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel