From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Mon, 29 Nov 2021 17:25:25 +0000 Subject: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful In-Reply-To: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> Message-ID: List-Id: To: kvm-riscv@lists.infradead.org MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit On Mon, Nov 29, 2021, Maxim Levitsky wrote: > (This thing is that when you tell the IOMMU that a vCPU is not running, > Another thing I discovered that this patch series totally breaks my VMs, > without cpu_pm=on The whole series (I didn't yet bisect it) makes even my > fedora32 VM be very laggy, almost unusable, and it only has one > passed-through device, a nic). Grrrr, the complete lack of comments in the KVM code and the separate paths for VMX vs SVM when handling HLT with APICv make this all way for difficult to understand than it should be. The hangs are likely due to: KVM: SVM: Unconditionally mark AVIC as running on vCPU load (with APICv) If a posted interrupt arrives after KVM has done its final search through the vIRR, but before avic_update_iommu_vcpu_affinity() is called, the posted interrupt will be set in the vIRR without triggering a host IRQ to wake the vCPU via the GA log. I.e. KVM is missing an equivalent to VMX's posted interrupt check for an outstanding notification after switching to the wakeup vector. For now, the least awful approach is sadly to keep the vcpu_(un)blocking() hooks. Unlike VMX's PI support, there's no fast check for an interrupt being posted (KVM would have to rewalk the vIRR), no easy to signal the current CPU to do wakeup (I don't think KVM even has access to the IRQ used by the owning IOMMU), and there's no simplification of load/put code. If the scheduler were changed to support waking in the sched_out path, then I'd be more inclined to handle this in avic_vcpu_put() by rewalking the vIRR one final time, but for now it's not worth it. > If I apply though only the patch series up to this patch, my fedora VM seems > to work fine, but my windows VM still locks up hard when I run 'LatencyTop' > in it, which doesn't happen without this patch. Buy "run 'LatencyTop' in it", do you mean running something in the Windows guest? The only search results I can find for LatencyTop are Linux specific. > So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt > (0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to > have different values), and IRR has plenty of interrupts with lower priority. > The VM seems to be stuck in this case. As if its EOI got lost or something is > preventing the IRQ handler from issuing EOI. > > LatencyTop does install some form of a kernel driver which likely does meddle > with interrupts (maybe it sends lots of self IPIs?). > > 100% reproducible as soon as I start monitoring with LatencyTop. > > Without this patch it works (or if disabling halt polling), Huh. I assume everything works if you disable halt polling _without_ this patch applied? If so, that implies that successful halt polling without mucking with vCPU IOMMU affinity is somehow problematic. I can't think of any relevant side effects other than timing. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Sean Christopherson Date: Mon, 29 Nov 2021 17:25:25 +0000 Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful Message-Id: List-Id: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> In-Reply-To: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit To: Maxim Levitsky Cc: Cornelia Huck , Wanpeng Li , kvm@vger.kernel.org, David Hildenbrand , linux-kernel@vger.kernel.org, Paul Mackerras , Atish Patra , linux-riscv@lists.infradead.org, Claudio Imbrenda , kvmarm@lists.cs.columbia.edu, Janosch Frank , Marc Zyngier , Joerg Roedel , Huacai Chen , Christian Borntraeger , Aleksandar Markovic , Albert Ou , kvm-ppc@vger.kernel.org, Paul Walmsley , David Matlack , linux-arm-kernel@lists.infradead.org, Jim Mattson , Anup Patel , linux-mips@vger.kernel.org, Palmer Dabbelt , kvm-riscv@lists.infradead.org, Paolo Bonzini , Vitaly Kuznetsov On Mon, Nov 29, 2021, Maxim Levitsky wrote: > (This thing is that when you tell the IOMMU that a vCPU is not running, > Another thing I discovered that this patch series totally breaks my VMs, > without cpu_pm=on The whole series (I didn't yet bisect it) makes even my > fedora32 VM be very laggy, almost unusable, and it only has one > passed-through device, a nic). Grrrr, the complete lack of comments in the KVM code and the separate paths for VMX vs SVM when handling HLT with APICv make this all way for difficult to understand than it should be. The hangs are likely due to: KVM: SVM: Unconditionally mark AVIC as running on vCPU load (with APICv) If a posted interrupt arrives after KVM has done its final search through the vIRR, but before avic_update_iommu_vcpu_affinity() is called, the posted interrupt will be set in the vIRR without triggering a host IRQ to wake the vCPU via the GA log. I.e. KVM is missing an equivalent to VMX's posted interrupt check for an outstanding notification after switching to the wakeup vector. For now, the least awful approach is sadly to keep the vcpu_(un)blocking() hooks. Unlike VMX's PI support, there's no fast check for an interrupt being posted (KVM would have to rewalk the vIRR), no easy to signal the current CPU to do wakeup (I don't think KVM even has access to the IRQ used by the owning IOMMU), and there's no simplification of load/put code. If the scheduler were changed to support waking in the sched_out path, then I'd be more inclined to handle this in avic_vcpu_put() by rewalking the vIRR one final time, but for now it's not worth it. > If I apply though only the patch series up to this patch, my fedora VM seems > to work fine, but my windows VM still locks up hard when I run 'LatencyTop' > in it, which doesn't happen without this patch. Buy "run 'LatencyTop' in it", do you mean running something in the Windows guest? The only search results I can find for LatencyTop are Linux specific. > So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt > (0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to > have different values), and IRR has plenty of interrupts with lower priority. > The VM seems to be stuck in this case. As if its EOI got lost or something is > preventing the IRQ handler from issuing EOI. > > LatencyTop does install some form of a kernel driver which likely does meddle > with interrupts (maybe it sends lots of self IPIs?). > > 100% reproducible as soon as I start monitoring with LatencyTop. > > Without this patch it works (or if disabling halt polling), Huh. I assume everything works if you disable halt polling _without_ this patch applied? If so, that implies that successful halt polling without mucking with vCPU IOMMU affinity is somehow problematic. I can't think of any relevant side effects other than timing. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from mm01.cs.columbia.edu (mm01.cs.columbia.edu [128.59.11.253]) by smtp.lore.kernel.org (Postfix) with ESMTP id E4717C433F5 for ; Mon, 29 Nov 2021 17:25:34 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 49CE14B16F; Mon, 29 Nov 2021 12:25:34 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Authentication-Results: mm01.cs.columbia.edu (amavisd-new); dkim=softfail (fail, message has been altered) header.i=@google.com Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id tP9APBxAVZIF; Mon, 29 Nov 2021 12:25:32 -0500 (EST) Received: from mm01.cs.columbia.edu (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id B5D1F4B166; Mon, 29 Nov 2021 12:25:32 -0500 (EST) Received: from localhost (localhost [127.0.0.1]) by mm01.cs.columbia.edu (Postfix) with ESMTP id 35A774B156 for ; Mon, 29 Nov 2021 12:25:32 -0500 (EST) X-Virus-Scanned: at lists.cs.columbia.edu Received: from mm01.cs.columbia.edu ([127.0.0.1]) by localhost (mm01.cs.columbia.edu [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id R8Og4mSi5SQa for ; Mon, 29 Nov 2021 12:25:30 -0500 (EST) Received: from mail-pl1-f176.google.com (mail-pl1-f176.google.com [209.85.214.176]) by mm01.cs.columbia.edu (Postfix) with ESMTPS id D1DBB4B0B9 for ; Mon, 29 Nov 2021 12:25:30 -0500 (EST) Received: by mail-pl1-f176.google.com with SMTP id y8so12743313plg.1 for ; Mon, 29 Nov 2021 09:25:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=OQsEdj0rbR41/5YmIUdSrIJYCBGgCTUy9MQ5b9N87MM6uHjwmzAsHY1InGAtcJzzT0 kXYHaRVbVnijPR96HzFaJq33Sw0nxu7KBR5alKkujlnpJgTb7ASLVjrwO1SSGsYoOkaQ K7cyD5kT9W1uMzNVmFhkVgYWGz7yTXUI7MuAi/f6xjABudDnAM1CxSfahAqvYF0vFr+z P1vOrLjB/9DR5TYDkZ2qVm7gvSexhNg5yeXAFAs7Vo4H0IrRkItMjbCy7SznqoFOYmJJ HNBdnQMeynqmiQ4BF5DB9MM0TzHLeCRZF1Krj6haC25keEJILJKSu+fr0lje3TcmjayD +Y/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=E8Xf56z83FHzD3P9B4evbWH7rCcinTNl1+9CRoUqT45JXdh7QtHPWk2Kbyip+tzN/Z wPcVCr49j/Ir3srHdW5VCRyJKS1v7zb/u2DtsHxg0BpavWawyAcG9Prn0x85VOFOcgKc CFRMJ8hQIm1ptdETzGSyCdCbmbIqOMzv3cV/Hujd6YG621nPd32ltCYPsRk2gD4Wewij pxJLopFmePwOzTOFU8yPcGcVNFDrQLqOqkX48p5wARpTW4YzZr5lTbqA+jZECjzXhRNz 72pVSWra1/CUuh+EdESniq9BZwPRIXn9Gk5M3d0fgV7254NpUykfT73BC8E0lVlq9/Ar AN2A== X-Gm-Message-State: AOAM533yfcDlfPksKb9Ihi6ms1dpji+GyhMNvDb1v4YEmocyJ4i4Mi+A bOIMjKCz9m8L5lC+lFeon6haqQ== X-Google-Smtp-Source: ABdhPJykBaUp6Ra73BKveHHRMZh4OnjUi7muGD2LnVC8BwpUUYIBB4aGQnyy+EYpnJ0P8dJ+mpdu6Q== X-Received: by 2002:a17:902:7289:b0:142:805f:e2c with SMTP id d9-20020a170902728900b00142805f0e2cmr61761710pll.42.1638206729695; Mon, 29 Nov 2021 09:25:29 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id m15sm18877279pjc.35.2021.11.29.09.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 09:25:29 -0800 (PST) Date: Mon, 29 Nov 2021 17:25:25 +0000 From: Sean Christopherson To: Maxim Levitsky Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> Cc: Cornelia Huck , Wanpeng Li , kvm@vger.kernel.org, David Hildenbrand , linux-kernel@vger.kernel.org, Paul Mackerras , Atish Patra , linux-riscv@lists.infradead.org, Claudio Imbrenda , kvmarm@lists.cs.columbia.edu, Janosch Frank , Marc Zyngier , Joerg Roedel , Huacai Chen , Christian Borntraeger , Aleksandar Markovic , Albert Ou , kvm-ppc@vger.kernel.org, Paul Walmsley , David Matlack , linux-arm-kernel@lists.infradead.org, Jim Mattson , Anup Patel , linux-mips@vger.kernel.org, Palmer Dabbelt , kvm-riscv@lists.infradead.org, Paolo Bonzini , Vitaly Kuznetsov X-BeenThere: kvmarm@lists.cs.columbia.edu X-Mailman-Version: 2.1.14 Precedence: list List-Id: Where KVM/ARM decisions are made List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: kvmarm-bounces@lists.cs.columbia.edu Sender: kvmarm-bounces@lists.cs.columbia.edu On Mon, Nov 29, 2021, Maxim Levitsky wrote: > (This thing is that when you tell the IOMMU that a vCPU is not running, > Another thing I discovered that this patch series totally breaks my VMs, > without cpu_pm=on The whole series (I didn't yet bisect it) makes even my > fedora32 VM be very laggy, almost unusable, and it only has one > passed-through device, a nic). Grrrr, the complete lack of comments in the KVM code and the separate paths for VMX vs SVM when handling HLT with APICv make this all way for difficult to understand than it should be. The hangs are likely due to: KVM: SVM: Unconditionally mark AVIC as running on vCPU load (with APICv) If a posted interrupt arrives after KVM has done its final search through the vIRR, but before avic_update_iommu_vcpu_affinity() is called, the posted interrupt will be set in the vIRR without triggering a host IRQ to wake the vCPU via the GA log. I.e. KVM is missing an equivalent to VMX's posted interrupt check for an outstanding notification after switching to the wakeup vector. For now, the least awful approach is sadly to keep the vcpu_(un)blocking() hooks. Unlike VMX's PI support, there's no fast check for an interrupt being posted (KVM would have to rewalk the vIRR), no easy to signal the current CPU to do wakeup (I don't think KVM even has access to the IRQ used by the owning IOMMU), and there's no simplification of load/put code. If the scheduler were changed to support waking in the sched_out path, then I'd be more inclined to handle this in avic_vcpu_put() by rewalking the vIRR one final time, but for now it's not worth it. > If I apply though only the patch series up to this patch, my fedora VM seems > to work fine, but my windows VM still locks up hard when I run 'LatencyTop' > in it, which doesn't happen without this patch. Buy "run 'LatencyTop' in it", do you mean running something in the Windows guest? The only search results I can find for LatencyTop are Linux specific. > So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt > (0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to > have different values), and IRR has plenty of interrupts with lower priority. > The VM seems to be stuck in this case. As if its EOI got lost or something is > preventing the IRQ handler from issuing EOI. > > LatencyTop does install some form of a kernel driver which likely does meddle > with interrupts (maybe it sends lots of self IPIs?). > > 100% reproducible as soon as I start monitoring with LatencyTop. > > Without this patch it works (or if disabling halt polling), Huh. I assume everything works if you disable halt polling _without_ this patch applied? If so, that implies that successful halt polling without mucking with vCPU IOMMU affinity is somehow problematic. I can't think of any relevant side effects other than timing. _______________________________________________ kvmarm mailing list kvmarm@lists.cs.columbia.edu https://lists.cs.columbia.edu/mailman/listinfo/kvmarm From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id BCE22C433FE for ; Mon, 29 Nov 2021 20:46:45 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S233077AbhK2UuD (ORCPT ); Mon, 29 Nov 2021 15:50:03 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:36320 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S231298AbhK2UsD (ORCPT ); Mon, 29 Nov 2021 15:48:03 -0500 Received: from mail-pl1-x62a.google.com (mail-pl1-x62a.google.com [IPv6:2607:f8b0:4864:20::62a]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8B45FC0698E6 for ; Mon, 29 Nov 2021 09:25:30 -0800 (PST) Received: by mail-pl1-x62a.google.com with SMTP id z6so12740796plk.6 for ; Mon, 29 Nov 2021 09:25:30 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=OQsEdj0rbR41/5YmIUdSrIJYCBGgCTUy9MQ5b9N87MM6uHjwmzAsHY1InGAtcJzzT0 kXYHaRVbVnijPR96HzFaJq33Sw0nxu7KBR5alKkujlnpJgTb7ASLVjrwO1SSGsYoOkaQ K7cyD5kT9W1uMzNVmFhkVgYWGz7yTXUI7MuAi/f6xjABudDnAM1CxSfahAqvYF0vFr+z P1vOrLjB/9DR5TYDkZ2qVm7gvSexhNg5yeXAFAs7Vo4H0IrRkItMjbCy7SznqoFOYmJJ HNBdnQMeynqmiQ4BF5DB9MM0TzHLeCRZF1Krj6haC25keEJILJKSu+fr0lje3TcmjayD +Y/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=NfX9AcDsYEmBblPKsqoTGfEZlKWKD0OXsSKiYFg7mIZVftdYZCdi4noKHzdJcIu2tZ wg+xmNCoG+coltHTM6F5EYOyltRTfjwKFwHRBahfRqccLZwsNIKu8NS/yom5yPIMebdz 2JUmon2OncD70eVI9Lw89YYmE1p1jiHWHvxPq/MDsN/hAmvriESz9keFvAWLxgSaNXlN p1bKCPMCcJsbSpwQhNKzK1G1GT9zvbBIvbmklMYLTAxiMg1Uf+HfuyCAxj9atWzlpnhl kjzNS0ZbKkOvKYquMIrXz1Vx6yHrDrqn4Ay2hD3mFI6b0zcfmSJK6sjILtt6xFjJHTup nBFQ== X-Gm-Message-State: AOAM531zlacLz83GCdOtrQOuPSm3Bvw7kfPyZtLyFaUz9EA1+Mda00NM r8cCTQ5gy/78DuXOyDmo3iE1eQ== X-Google-Smtp-Source: ABdhPJykBaUp6Ra73BKveHHRMZh4OnjUi7muGD2LnVC8BwpUUYIBB4aGQnyy+EYpnJ0P8dJ+mpdu6Q== X-Received: by 2002:a17:902:7289:b0:142:805f:e2c with SMTP id d9-20020a170902728900b00142805f0e2cmr61761710pll.42.1638206729695; Mon, 29 Nov 2021 09:25:29 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id m15sm18877279pjc.35.2021.11.29.09.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 09:25:29 -0800 (PST) Date: Mon, 29 Nov 2021 17:25:25 +0000 From: Sean Christopherson To: Maxim Levitsky Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini , James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> Precedence: bulk List-ID: X-Mailing-List: linux-mips@vger.kernel.org On Mon, Nov 29, 2021, Maxim Levitsky wrote: > (This thing is that when you tell the IOMMU that a vCPU is not running, > Another thing I discovered that this patch series totally breaks my VMs, > without cpu_pm=on The whole series (I didn't yet bisect it) makes even my > fedora32 VM be very laggy, almost unusable, and it only has one > passed-through device, a nic). Grrrr, the complete lack of comments in the KVM code and the separate paths for VMX vs SVM when handling HLT with APICv make this all way for difficult to understand than it should be. The hangs are likely due to: KVM: SVM: Unconditionally mark AVIC as running on vCPU load (with APICv) If a posted interrupt arrives after KVM has done its final search through the vIRR, but before avic_update_iommu_vcpu_affinity() is called, the posted interrupt will be set in the vIRR without triggering a host IRQ to wake the vCPU via the GA log. I.e. KVM is missing an equivalent to VMX's posted interrupt check for an outstanding notification after switching to the wakeup vector. For now, the least awful approach is sadly to keep the vcpu_(un)blocking() hooks. Unlike VMX's PI support, there's no fast check for an interrupt being posted (KVM would have to rewalk the vIRR), no easy to signal the current CPU to do wakeup (I don't think KVM even has access to the IRQ used by the owning IOMMU), and there's no simplification of load/put code. If the scheduler were changed to support waking in the sched_out path, then I'd be more inclined to handle this in avic_vcpu_put() by rewalking the vIRR one final time, but for now it's not worth it. > If I apply though only the patch series up to this patch, my fedora VM seems > to work fine, but my windows VM still locks up hard when I run 'LatencyTop' > in it, which doesn't happen without this patch. Buy "run 'LatencyTop' in it", do you mean running something in the Windows guest? The only search results I can find for LatencyTop are Linux specific. > So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt > (0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to > have different values), and IRR has plenty of interrupts with lower priority. > The VM seems to be stuck in this case. As if its EOI got lost or something is > preventing the IRQ handler from issuing EOI. > > LatencyTop does install some form of a kernel driver which likely does meddle > with interrupts (maybe it sends lots of self IPIs?). > > 100% reproducible as soon as I start monitoring with LatencyTop. > > Without this patch it works (or if disabling halt polling), Huh. I assume everything works if you disable halt polling _without_ this patch applied? If so, that implies that successful halt polling without mucking with vCPU IOMMU affinity is somehow problematic. I can't think of any relevant side effects other than timing. From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 91F6FC433F5 for ; Mon, 29 Nov 2021 17:50:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=wY6COcsz1wRJkYEQQm1c9yHhDrtk22a1XlsXVcY3G+M=; b=uU+2IGDhhvJyUn p8hhKmtewCp2G75y6e9pDwQ2wMYR2xDa19LRECZzfheyrwTXmTw/ENSl0Up410CQxwllqzGBEx8Zi TTMQYdZnublSBXhPZt3pyprBjgp4qd+rYC+GZ/Aq4Z9J7bsc7ySpjPHyzGr9WS5SBgcWKwckKVb4Q +w9U53nl4jOzIibu/n6zVUESI+oEAfYkItn5eDOjek8kPq7DHeJPS39Af+oS5xDieBsEnxELNvfsV eXmLULveRfFHUSIFN6w/4spA0X24j24ex4vvUta0jBkEHXI+XjgQPpwzK6DcKM5uy5UDGug0lSGMN HWfmtZVTY3gVjWo1hW4Q==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrkmt-001df4-EJ; Mon, 29 Nov 2021 17:49:59 +0000 Received: from mail-pj1-x1034.google.com ([2607:f8b0:4864:20::1034]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrkmr-001dcf-03 for linux-riscv@lists.infradead.org; Mon, 29 Nov 2021 17:49:58 +0000 Received: by mail-pj1-x1034.google.com with SMTP id y14-20020a17090a2b4e00b001a5824f4918so16180517pjc.4 for ; Mon, 29 Nov 2021 09:49:53 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=OQsEdj0rbR41/5YmIUdSrIJYCBGgCTUy9MQ5b9N87MM6uHjwmzAsHY1InGAtcJzzT0 kXYHaRVbVnijPR96HzFaJq33Sw0nxu7KBR5alKkujlnpJgTb7ASLVjrwO1SSGsYoOkaQ K7cyD5kT9W1uMzNVmFhkVgYWGz7yTXUI7MuAi/f6xjABudDnAM1CxSfahAqvYF0vFr+z P1vOrLjB/9DR5TYDkZ2qVm7gvSexhNg5yeXAFAs7Vo4H0IrRkItMjbCy7SznqoFOYmJJ HNBdnQMeynqmiQ4BF5DB9MM0TzHLeCRZF1Krj6haC25keEJILJKSu+fr0lje3TcmjayD +Y/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=GGtGYryuZD7rA8E3LMt+P6dXQ9BUdN1/kmATu5aJVKyB7su9YMY1/UB1Gt0Fm4o/gw L/IuSZLEWm4LwJc7jU8Ifx3kkVtqCXvjI32MNK02H550iftLthcGFq6mV3aNLRVjz/Gk 3KD+UGxzMUz9TLgc1YVa1bWHORDwHMMqT7Oc7yhAjxipupZr1XjDwpB8fjXjMqEfQvwM kp40q66lZ2+iCKRX5so6JsIl53TkMcc83RMooFhloho7m/dcz3MumRc4Gz2MS99swbvM FP71IpwbsdE5YRr6WCxZTnuHKu5SA6ij7L9uNJMtkEXbGHJ/ZJpB9OZW/fuReK5x01RE GMLA== X-Gm-Message-State: AOAM530nTiMhFdT0B/uuphId6VAoglycsw7BJRpAYQYmfzpQcPNQ0own dATkU7QOtH7lWVDKN3l56aSIppHtC/YJ4Q== X-Google-Smtp-Source: ABdhPJykBaUp6Ra73BKveHHRMZh4OnjUi7muGD2LnVC8BwpUUYIBB4aGQnyy+EYpnJ0P8dJ+mpdu6Q== X-Received: by 2002:a17:902:7289:b0:142:805f:e2c with SMTP id d9-20020a170902728900b00142805f0e2cmr61761710pll.42.1638206729695; Mon, 29 Nov 2021 09:25:29 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id m15sm18877279pjc.35.2021.11.29.09.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 09:25:29 -0800 (PST) Date: Mon, 29 Nov 2021 17:25:25 +0000 From: Sean Christopherson To: Maxim Levitsky Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini , James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211129_094957_067272_67E187EC X-CRM114-Status: GOOD ( 24.52 ) X-BeenThere: linux-riscv@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-riscv" Errors-To: linux-riscv-bounces+linux-riscv=archiver.kernel.org@lists.infradead.org On Mon, Nov 29, 2021, Maxim Levitsky wrote: > (This thing is that when you tell the IOMMU that a vCPU is not running, > Another thing I discovered that this patch series totally breaks my VMs, > without cpu_pm=on The whole series (I didn't yet bisect it) makes even my > fedora32 VM be very laggy, almost unusable, and it only has one > passed-through device, a nic). Grrrr, the complete lack of comments in the KVM code and the separate paths for VMX vs SVM when handling HLT with APICv make this all way for difficult to understand than it should be. The hangs are likely due to: KVM: SVM: Unconditionally mark AVIC as running on vCPU load (with APICv) If a posted interrupt arrives after KVM has done its final search through the vIRR, but before avic_update_iommu_vcpu_affinity() is called, the posted interrupt will be set in the vIRR without triggering a host IRQ to wake the vCPU via the GA log. I.e. KVM is missing an equivalent to VMX's posted interrupt check for an outstanding notification after switching to the wakeup vector. For now, the least awful approach is sadly to keep the vcpu_(un)blocking() hooks. Unlike VMX's PI support, there's no fast check for an interrupt being posted (KVM would have to rewalk the vIRR), no easy to signal the current CPU to do wakeup (I don't think KVM even has access to the IRQ used by the owning IOMMU), and there's no simplification of load/put code. If the scheduler were changed to support waking in the sched_out path, then I'd be more inclined to handle this in avic_vcpu_put() by rewalking the vIRR one final time, but for now it's not worth it. > If I apply though only the patch series up to this patch, my fedora VM seems > to work fine, but my windows VM still locks up hard when I run 'LatencyTop' > in it, which doesn't happen without this patch. Buy "run 'LatencyTop' in it", do you mean running something in the Windows guest? The only search results I can find for LatencyTop are Linux specific. > So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt > (0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to > have different values), and IRR has plenty of interrupts with lower priority. > The VM seems to be stuck in this case. As if its EOI got lost or something is > preventing the IRQ handler from issuing EOI. > > LatencyTop does install some form of a kernel driver which likely does meddle > with interrupts (maybe it sends lots of self IPIs?). > > 100% reproducible as soon as I start monitoring with LatencyTop. > > Without this patch it works (or if disabling halt polling), Huh. I assume everything works if you disable halt polling _without_ this patch applied? If so, that implies that successful halt polling without mucking with vCPU IOMMU affinity is somehow problematic. I can't think of any relevant side effects other than timing. _______________________________________________ linux-riscv mailing list linux-riscv@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-riscv From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 4FD29C433FE for ; Mon, 29 Nov 2021 17:58:40 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=1M8nOPKaypK3k4+d+ldE4uJcSgeHsq2UsT2zJWvHpNU=; b=whdJOUiQudAtK+ 4zE9a0LCfdhXpU4hTfDPxVqV4OhGv+5uy9PkrxBIcgLLLIwJkPDox4DbTva3/JhFOt6Rb/iI/R68w QalKCm0S+4MYbPa4hF5TEYOsMU0cQ4b/XYkqwyrOYVEvaB8eU15GAfpZGqFEmziZTvS7p710q8GXu mKLsgiRu+7VCuSiBzOvnc2cKZ31w6tdzRoVhB6YFT+biBdoP26FyvqxlqnGN+HVdM39qSye2VHvDs F54A7M3kz1cWKT7T0GNIYFZ4uH6XM1GgALwlmsgVbWCkxYR589QXAdYcovkYyraahycIMeQKEPL3M g5UuQy8XlTUPlLiSJs3A==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrktu-001ftl-SL; Mon, 29 Nov 2021 17:57:15 +0000 Received: from mail-qv1-xf35.google.com ([2607:f8b0:4864:20::f35]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1mrktq-001fso-Rr for linux-arm-kernel@lists.infradead.org; Mon, 29 Nov 2021 17:57:12 +0000 Received: by mail-qv1-xf35.google.com with SMTP id jo22so15321409qvb.13 for ; Mon, 29 Nov 2021 09:57:10 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=OQsEdj0rbR41/5YmIUdSrIJYCBGgCTUy9MQ5b9N87MM6uHjwmzAsHY1InGAtcJzzT0 kXYHaRVbVnijPR96HzFaJq33Sw0nxu7KBR5alKkujlnpJgTb7ASLVjrwO1SSGsYoOkaQ K7cyD5kT9W1uMzNVmFhkVgYWGz7yTXUI7MuAi/f6xjABudDnAM1CxSfahAqvYF0vFr+z P1vOrLjB/9DR5TYDkZ2qVm7gvSexhNg5yeXAFAs7Vo4H0IrRkItMjbCy7SznqoFOYmJJ HNBdnQMeynqmiQ4BF5DB9MM0TzHLeCRZF1Krj6haC25keEJILJKSu+fr0lje3TcmjayD +Y/g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=Bxpm877ucu29+swtwCcWrV9CPuVdsV2sIZfuT/jfRmc=; b=TuWazjIz0FwGUaXCCKmxCABnPlMQ6G+McSuz+RbCjKx/mExru6c05iiZ0IwtAV358w jQctmaoMCwr7l74fJAkLk2UjlPpS73hF8WTD1zohaHXIdI0g5kD1OD1NkNPP4WHerujf q4p6fKOo2QrmTkqXEz+3Bl6cRLrig8beNKDba15mFHoHeAJ9aqna6JO6lyWtmkqRnK8s dljMO2Pwqh42RZj3ayaty0atw0uX9zNQCJuALBXbx2gkhHk2UjJArZn8BK59MgHziHVF zR1hFoDfQv8ADjw7zxG4wz2iUNb+yFNqPbWvhCgtqGMT50nE95OYRR4IoMRdK9QptMkf cd9A== X-Gm-Message-State: AOAM532y/atuogvt13apxbrUAJ6/2QFkuhhWKD++kQVgLzWYZToqeQNq lgFONPjAeqQfRtmMoNvn44jf4GURC7bQIQ== X-Google-Smtp-Source: ABdhPJykBaUp6Ra73BKveHHRMZh4OnjUi7muGD2LnVC8BwpUUYIBB4aGQnyy+EYpnJ0P8dJ+mpdu6Q== X-Received: by 2002:a17:902:7289:b0:142:805f:e2c with SMTP id d9-20020a170902728900b00142805f0e2cmr61761710pll.42.1638206729695; Mon, 29 Nov 2021 09:25:29 -0800 (PST) Received: from google.com (157.214.185.35.bc.googleusercontent.com. [35.185.214.157]) by smtp.gmail.com with ESMTPSA id m15sm18877279pjc.35.2021.11.29.09.25.28 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 29 Nov 2021 09:25:29 -0800 (PST) Date: Mon, 29 Nov 2021 17:25:25 +0000 From: Sean Christopherson To: Maxim Levitsky Cc: Marc Zyngier , Huacai Chen , Aleksandar Markovic , Paul Mackerras , Anup Patel , Paul Walmsley , Palmer Dabbelt , Albert Ou , Christian Borntraeger , Janosch Frank , Paolo Bonzini , James Morse , Alexandru Elisei , Suzuki K Poulose , Atish Patra , David Hildenbrand , Cornelia Huck , Claudio Imbrenda , Vitaly Kuznetsov , Wanpeng Li , Jim Mattson , Joerg Roedel , linux-arm-kernel@lists.infradead.org, kvmarm@lists.cs.columbia.edu, linux-mips@vger.kernel.org, kvm@vger.kernel.org, kvm-ppc@vger.kernel.org, kvm-riscv@lists.infradead.org, linux-riscv@lists.infradead.org, linux-kernel@vger.kernel.org, David Matlack , Oliver Upton , Jing Zhang Subject: Re: [PATCH v2 11/43] KVM: Don't block+unblock when halt-polling is successful Message-ID: References: <20211009021236.4122790-1-seanjc@google.com> <20211009021236.4122790-12-seanjc@google.com> <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <4e883728e3e5201a94eb46b56315afca5e95ad9c.camel@redhat.com> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20211129_095710_918554_18C00635 X-CRM114-Status: GOOD ( 25.92 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Mon, Nov 29, 2021, Maxim Levitsky wrote: > (This thing is that when you tell the IOMMU that a vCPU is not running, > Another thing I discovered that this patch series totally breaks my VMs, > without cpu_pm=on The whole series (I didn't yet bisect it) makes even my > fedora32 VM be very laggy, almost unusable, and it only has one > passed-through device, a nic). Grrrr, the complete lack of comments in the KVM code and the separate paths for VMX vs SVM when handling HLT with APICv make this all way for difficult to understand than it should be. The hangs are likely due to: KVM: SVM: Unconditionally mark AVIC as running on vCPU load (with APICv) If a posted interrupt arrives after KVM has done its final search through the vIRR, but before avic_update_iommu_vcpu_affinity() is called, the posted interrupt will be set in the vIRR without triggering a host IRQ to wake the vCPU via the GA log. I.e. KVM is missing an equivalent to VMX's posted interrupt check for an outstanding notification after switching to the wakeup vector. For now, the least awful approach is sadly to keep the vcpu_(un)blocking() hooks. Unlike VMX's PI support, there's no fast check for an interrupt being posted (KVM would have to rewalk the vIRR), no easy to signal the current CPU to do wakeup (I don't think KVM even has access to the IRQ used by the owning IOMMU), and there's no simplification of load/put code. If the scheduler were changed to support waking in the sched_out path, then I'd be more inclined to handle this in avic_vcpu_put() by rewalking the vIRR one final time, but for now it's not worth it. > If I apply though only the patch series up to this patch, my fedora VM seems > to work fine, but my windows VM still locks up hard when I run 'LatencyTop' > in it, which doesn't happen without this patch. Buy "run 'LatencyTop' in it", do you mean running something in the Windows guest? The only search results I can find for LatencyTop are Linux specific. > So far the symptoms I see is that on VCPU 0, ISR has quite high interrupt > (0xe1 last time I seen it), TPR and PPR are 0xe0 (although I have seen TPR to > have different values), and IRR has plenty of interrupts with lower priority. > The VM seems to be stuck in this case. As if its EOI got lost or something is > preventing the IRQ handler from issuing EOI. > > LatencyTop does install some form of a kernel driver which likely does meddle > with interrupts (maybe it sends lots of self IPIs?). > > 100% reproducible as soon as I start monitoring with LatencyTop. > > Without this patch it works (or if disabling halt polling), Huh. I assume everything works if you disable halt polling _without_ this patch applied? If so, that implies that successful halt polling without mucking with vCPU IOMMU affinity is somehow problematic. I can't think of any relevant side effects other than timing. _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel