From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 5C477C433F5 for ; Mon, 3 Oct 2022 14:25:06 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id C4A8B10E3D4; Mon, 3 Oct 2022 14:25:04 +0000 (UTC) Received: from mail-pj1-x1036.google.com (mail-pj1-x1036.google.com [IPv6:2607:f8b0:4864:20::1036]) by gabe.freedesktop.org (Postfix) with ESMTPS id 8434410EB9A for ; Thu, 29 Sep 2022 22:38:10 +0000 (UTC) Received: by mail-pj1-x1036.google.com with SMTP id u59-20020a17090a51c100b00205d3c44162so7307816pjh.2 for ; Thu, 29 Sep 2022 15:38:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date; bh=uRYofiYnG47hAF2aDgziR8isJuYRbsZpgovud56j32A=; b=JZPVhVcSMWJOlPWPeuO7TIk2QE8xzCYuAuPJMIvNL9e/7DEG7nT1s/j0a++TuIXoJt W5tOkUQ3APGpHGPx2aDs0EQFmPuIgBA+4T0avm1Lg0UJhYZ3xUC1RzFzgaGhbslD+zEA 8NgZgevTkenFB0f0XOWYHbeCZsUerfUuHJ0MRFMwXM5XuUFT0t5SADRdJ0rMU3iZnQpx iMve53x366pyZVcyz/wlrlsM3/1JBISjrdscSFwoyHMKAyPnzbLjIdZA8bJqQmwrTVBh qX5XlvQpPaLRv1P2ont+Bp7rvzNsL9me5BwcB/JytVGVWSrVmlRW7tys/zv1TCWNsOeo isHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date; bh=uRYofiYnG47hAF2aDgziR8isJuYRbsZpgovud56j32A=; b=74zptF+SXL0pQ/8phWPvtYygAINIayh+oVBymKfTLcpWkiSbBRgtftZVFKdM5/v9K3 xxmjj71g5k4EeicjtgjE0CNRaV4X7IZutgEyMGMBl2W8rN6sR/nwoQ8dGn1ecWYT1X0j ujbzVswKW5h1BprFd3rZDNK/I/jDWqIPg6a31kWazVFy1y24yCXojjtlwpjad40dxLy2 RfFuZotg5d6/4n4uo3/cAqY+XpLti9sDV4Xc/1rAbCWVkntKznvMBnfaV9x9AztAMmLa mb5EpaMy+DZpeCUb9Hj/0lAIzlxbw42csYYxKa6u+QG3+VUUNeWJPMmjEwQ25HiiBaT/ evuA== X-Gm-Message-State: ACrzQf1mPalazJtugsFAt7guuBr7LIbmdo8pkegeQ0/SlhltlTpGk9yD 15FLkVuiIQqTPS2Mx+R/6VarwQ== X-Google-Smtp-Source: AMsMyM6yC6NkaBuBd+HCYZGGqRfzh1aqwwfKXWPEPMxkZ/E1X0XuzRLSKCq/ziAdT/GnArgh0pqgvg== X-Received: by 2002:a17:903:441:b0:179:f1cc:ba89 with SMTP id iw1-20020a170903044100b00179f1ccba89mr5683259plb.146.1664491089036; Thu, 29 Sep 2022 15:38:09 -0700 (PDT) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id w2-20020a170902e88200b0017829a3df46sm384797plg.204.2022.09.29.15.38.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Sep 2022 15:38:08 -0700 (PDT) Date: Thu, 29 Sep 2022 22:38:04 +0000 From: Sean Christopherson To: Maxim Levitsky Message-ID: References: <20220427200314.276673-1-mlevitsk@redhat.com> <20220427200314.276673-5-mlevitsk@redhat.com> <5ed0d0e5a88bbee2f95d794dbbeb1ad16789f319.camel@redhat.com> <7c4cf32dca42ab84bdb427a9e4862dbf5509f961.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Mailman-Approved-At: Mon, 03 Oct 2022 14:24:59 +0000 Subject: Re: [Intel-gfx] Nested AVIC design (was:Re: [RFC PATCH v3 04/19] KVM: x86: mmu: allow to enable write tracking externally) X-BeenThere: intel-gfx@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Intel graphics driver community testing & development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Wanpeng Li , kvm@vger.kernel.org, David Airlie , Dave Hansen , dri-devel@lists.freedesktop.org, "H. Peter Anvin" , Brijesh Singh , Joerg Roedel , x86@kernel.org, Ingo Molnar , Tom Lendacky , intel-gfx@lists.freedesktop.org, Borislav Petkov , Rodrigo Vivi , Thomas Gleixner , intel-gvt-dev@lists.freedesktop.org, Jim Mattson , linux-kernel@vger.kernel.org, Daniel Vetter , Paolo Bonzini , Vitaly Kuznetsov Errors-To: intel-gfx-bounces@lists.freedesktop.org Sender: "Intel-gfx" On Mon, Aug 08, 2022, Maxim Levitsky wrote: > Hi Sean, Paolo, and everyone else who wants to review my nested AVIC work. Before we dive deep into design details, I think we should first decide whether or not nested AVIC is worth pursing/supporting. - Rome has a ucode/silicon bug with no known workaround and no anticipated fix[*]; AMD's recommended "workaround" is to disable AVIC. - AVIC is not available in Milan, which may or may not be related to the aforementioned bug. - AVIC is making a comeback on Zen4, but Zen4 comes with x2AVIC. - x2APIC is likely going to become ubiquitous, e.g. Intel is effectively requiring x2APIC to fudge around xAPIC bugs. - It's actually quite realistic to effectively force the guest to use x2APIC, at least if it's a Linux guest. E.g. turn x2APIC on in BIOS, which is often (always?) controlled by the host, and Linux will use x2APIC. In other words, given that AVIC is well on its way to becoming a "legacy" feature, IMO there needs to be a fairly strong use case to justify taking on this much code and complexity. ~1500 lines of code to support a feature that has historically been buggy _without_ nested support is going to require a non-trivial amount of effort to review, stabilize, and maintain. [*] 1235 "Guest With AVIC (Advanced Virtual Interrupt Controller) Enabled May Fail to Process IPI (Inter-Processor Interrupt) Until Guest Is Re-Scheduled" in https://www.amd.com/system/files/TechDocs/56323-PUB_1.00.pdf From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from gabe.freedesktop.org (gabe.freedesktop.org [131.252.210.177]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id B7DFDC433F5 for ; Thu, 29 Sep 2022 22:38:17 +0000 (UTC) Received: from gabe.freedesktop.org (localhost [127.0.0.1]) by gabe.freedesktop.org (Postfix) with ESMTP id E3D5210E146; Thu, 29 Sep 2022 22:38:15 +0000 (UTC) Received: from mail-pj1-x1033.google.com (mail-pj1-x1033.google.com [IPv6:2607:f8b0:4864:20::1033]) by gabe.freedesktop.org (Postfix) with ESMTPS id 2A2C210E18D for ; Thu, 29 Sep 2022 22:38:10 +0000 (UTC) Received: by mail-pj1-x1033.google.com with SMTP id u92so2630017pjh.3 for ; Thu, 29 Sep 2022 15:38:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date; bh=uRYofiYnG47hAF2aDgziR8isJuYRbsZpgovud56j32A=; b=JZPVhVcSMWJOlPWPeuO7TIk2QE8xzCYuAuPJMIvNL9e/7DEG7nT1s/j0a++TuIXoJt W5tOkUQ3APGpHGPx2aDs0EQFmPuIgBA+4T0avm1Lg0UJhYZ3xUC1RzFzgaGhbslD+zEA 8NgZgevTkenFB0f0XOWYHbeCZsUerfUuHJ0MRFMwXM5XuUFT0t5SADRdJ0rMU3iZnQpx iMve53x366pyZVcyz/wlrlsM3/1JBISjrdscSFwoyHMKAyPnzbLjIdZA8bJqQmwrTVBh qX5XlvQpPaLRv1P2ont+Bp7rvzNsL9me5BwcB/JytVGVWSrVmlRW7tys/zv1TCWNsOeo isHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date; bh=uRYofiYnG47hAF2aDgziR8isJuYRbsZpgovud56j32A=; b=Q67F3SUBZNeM0G7UnquaBJoM3nc8dwtPOsHTchb7sy8px5GujzFtEHH2djlOb19j9w TJkRkSYPD6bdfqAJ+gAJ5AVdnJlJSunlvnkK+dhmnIie1yCoA6k1+aKXCy8vajOpHbi7 tZq1GEXleY4plJpTKkN+XiKPlv5lm5ZJ8BLZAMW/C31dODR91Dw/+AH/WxVn+9EXm0mO Rqw+JLyCM7A6FfgpI/eFD2V8XVD4E/nHA9C4rGPmLuQNkYRTucGR6tQO9DOSvbmX7gyR b02Hjl0eqsLDyDasP6zfiuhn9P7FkZa0WbaTZQ2KR/tl4FR2Yc5wl9oFngmCKQF8Zr9n 8uMA== X-Gm-Message-State: ACrzQf2ERp3AdxJXv8RyMemFTeaRr+uneUaWLExDn4cFByxE0uHkT3wQ NNeM0cV6NZALE5QO87PTRcQdbA== X-Google-Smtp-Source: AMsMyM6yC6NkaBuBd+HCYZGGqRfzh1aqwwfKXWPEPMxkZ/E1X0XuzRLSKCq/ziAdT/GnArgh0pqgvg== X-Received: by 2002:a17:903:441:b0:179:f1cc:ba89 with SMTP id iw1-20020a170903044100b00179f1ccba89mr5683259plb.146.1664491089036; Thu, 29 Sep 2022 15:38:09 -0700 (PDT) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id w2-20020a170902e88200b0017829a3df46sm384797plg.204.2022.09.29.15.38.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Sep 2022 15:38:08 -0700 (PDT) Date: Thu, 29 Sep 2022 22:38:04 +0000 From: Sean Christopherson To: Maxim Levitsky Subject: Re: Nested AVIC design (was:Re: [RFC PATCH v3 04/19] KVM: x86: mmu: allow to enable write tracking externally) Message-ID: References: <20220427200314.276673-1-mlevitsk@redhat.com> <20220427200314.276673-5-mlevitsk@redhat.com> <5ed0d0e5a88bbee2f95d794dbbeb1ad16789f319.camel@redhat.com> <7c4cf32dca42ab84bdb427a9e4862dbf5509f961.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-BeenThere: dri-devel@lists.freedesktop.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: Direct Rendering Infrastructure - Development List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Wanpeng Li , kvm@vger.kernel.org, David Airlie , Dave Hansen , dri-devel@lists.freedesktop.org, "H. Peter Anvin" , Brijesh Singh , Joerg Roedel , x86@kernel.org, Ingo Molnar , Zhi Wang , Tom Lendacky , intel-gfx@lists.freedesktop.org, Borislav Petkov , Rodrigo Vivi , Thomas Gleixner , intel-gvt-dev@lists.freedesktop.org, Jim Mattson , Tvrtko Ursulin , linux-kernel@vger.kernel.org, Paolo Bonzini , Vitaly Kuznetsov Errors-To: dri-devel-bounces@lists.freedesktop.org Sender: "dri-devel" On Mon, Aug 08, 2022, Maxim Levitsky wrote: > Hi Sean, Paolo, and everyone else who wants to review my nested AVIC work. Before we dive deep into design details, I think we should first decide whether or not nested AVIC is worth pursing/supporting. - Rome has a ucode/silicon bug with no known workaround and no anticipated fix[*]; AMD's recommended "workaround" is to disable AVIC. - AVIC is not available in Milan, which may or may not be related to the aforementioned bug. - AVIC is making a comeback on Zen4, but Zen4 comes with x2AVIC. - x2APIC is likely going to become ubiquitous, e.g. Intel is effectively requiring x2APIC to fudge around xAPIC bugs. - It's actually quite realistic to effectively force the guest to use x2APIC, at least if it's a Linux guest. E.g. turn x2APIC on in BIOS, which is often (always?) controlled by the host, and Linux will use x2APIC. In other words, given that AVIC is well on its way to becoming a "legacy" feature, IMO there needs to be a fairly strong use case to justify taking on this much code and complexity. ~1500 lines of code to support a feature that has historically been buggy _without_ nested support is going to require a non-trivial amount of effort to review, stabilize, and maintain. [*] 1235 "Guest With AVIC (Advanced Virtual Interrupt Controller) Enabled May Fail to Process IPI (Inter-Processor Interrupt) Until Guest Is Re-Scheduled" in https://www.amd.com/system/files/TechDocs/56323-PUB_1.00.pdf From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AC45C433F5 for ; Thu, 29 Sep 2022 22:44:00 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230019AbiI2Wn6 (ORCPT ); Thu, 29 Sep 2022 18:43:58 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:32784 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S229506AbiI2WnI (ORCPT ); Thu, 29 Sep 2022 18:43:08 -0400 Received: from mail-pj1-x1035.google.com (mail-pj1-x1035.google.com [IPv6:2607:f8b0:4864:20::1035]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 8FA14184817 for ; Thu, 29 Sep 2022 15:38:46 -0700 (PDT) Received: by mail-pj1-x1035.google.com with SMTP id x1-20020a17090ab00100b001fda21bbc90so7292168pjq.3 for ; Thu, 29 Sep 2022 15:38:46 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date; bh=uRYofiYnG47hAF2aDgziR8isJuYRbsZpgovud56j32A=; b=JZPVhVcSMWJOlPWPeuO7TIk2QE8xzCYuAuPJMIvNL9e/7DEG7nT1s/j0a++TuIXoJt W5tOkUQ3APGpHGPx2aDs0EQFmPuIgBA+4T0avm1Lg0UJhYZ3xUC1RzFzgaGhbslD+zEA 8NgZgevTkenFB0f0XOWYHbeCZsUerfUuHJ0MRFMwXM5XuUFT0t5SADRdJ0rMU3iZnQpx iMve53x366pyZVcyz/wlrlsM3/1JBISjrdscSFwoyHMKAyPnzbLjIdZA8bJqQmwrTVBh qX5XlvQpPaLRv1P2ont+Bp7rvzNsL9me5BwcB/JytVGVWSrVmlRW7tys/zv1TCWNsOeo isHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date; bh=uRYofiYnG47hAF2aDgziR8isJuYRbsZpgovud56j32A=; b=NSgAPzqu4HLnDHqzVuOVKxw1QNpVfN+Fq0baYTTVu26TqHLhk6rI8EAe7h8ZRtm69w giyQMV4SLdX4bSL3rFelKAm+yQRZNwAUBX0Zog966DqC2hnD3QhZI5ymTp7fSJI9i3m0 3yYJnJTx4nkPt+UeMeaIsXkT3YgiVUTztfE7jDAYx7q86HtcphEzQfS7YSr9h4pGWUOF h5U4cJuqS3/VDF/KrVjO9qyTNjJizcGQmqBU2cpUk2+V+dY+GJPtfOuTo63NyZx+zQ7T cVzzlLoCc1eznSOfsVOBsaxDk0o3gBlG+g1hQJr24O/Ndb74cibpOo850/d59qsOYgvr jY4Q== X-Gm-Message-State: ACrzQf1559TUbm0VgjUqV8NGcSSuVC+OWoouDsPMhtrDIQa7AyY4nCVB dG7/GdvkEGWHfUMv8jiSl+KWSg== X-Google-Smtp-Source: AMsMyM6yC6NkaBuBd+HCYZGGqRfzh1aqwwfKXWPEPMxkZ/E1X0XuzRLSKCq/ziAdT/GnArgh0pqgvg== X-Received: by 2002:a17:903:441:b0:179:f1cc:ba89 with SMTP id iw1-20020a170903044100b00179f1ccba89mr5683259plb.146.1664491089036; Thu, 29 Sep 2022 15:38:09 -0700 (PDT) Received: from google.com (7.104.168.34.bc.googleusercontent.com. [34.168.104.7]) by smtp.gmail.com with ESMTPSA id w2-20020a170902e88200b0017829a3df46sm384797plg.204.2022.09.29.15.38.08 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 29 Sep 2022 15:38:08 -0700 (PDT) Date: Thu, 29 Sep 2022 22:38:04 +0000 From: Sean Christopherson To: Maxim Levitsky Cc: kvm@vger.kernel.org, Wanpeng Li , Vitaly Kuznetsov , Jani Nikula , Paolo Bonzini , Tvrtko Ursulin , Rodrigo Vivi , Zhenyu Wang , Joonas Lahtinen , Tom Lendacky , Ingo Molnar , David Airlie , Thomas Gleixner , Dave Hansen , x86@kernel.org, intel-gfx@lists.freedesktop.org, Daniel Vetter , Borislav Petkov , Joerg Roedel , linux-kernel@vger.kernel.org, Jim Mattson , Zhi Wang , Brijesh Singh , "H. Peter Anvin" , intel-gvt-dev@lists.freedesktop.org, dri-devel@lists.freedesktop.org Subject: Re: Nested AVIC design (was:Re: [RFC PATCH v3 04/19] KVM: x86: mmu: allow to enable write tracking externally) Message-ID: References: <20220427200314.276673-1-mlevitsk@redhat.com> <20220427200314.276673-5-mlevitsk@redhat.com> <5ed0d0e5a88bbee2f95d794dbbeb1ad16789f319.camel@redhat.com> <7c4cf32dca42ab84bdb427a9e4862dbf5509f961.camel@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Precedence: bulk List-ID: X-Mailing-List: kvm@vger.kernel.org On Mon, Aug 08, 2022, Maxim Levitsky wrote: > Hi Sean, Paolo, and everyone else who wants to review my nested AVIC work. Before we dive deep into design details, I think we should first decide whether or not nested AVIC is worth pursing/supporting. - Rome has a ucode/silicon bug with no known workaround and no anticipated fix[*]; AMD's recommended "workaround" is to disable AVIC. - AVIC is not available in Milan, which may or may not be related to the aforementioned bug. - AVIC is making a comeback on Zen4, but Zen4 comes with x2AVIC. - x2APIC is likely going to become ubiquitous, e.g. Intel is effectively requiring x2APIC to fudge around xAPIC bugs. - It's actually quite realistic to effectively force the guest to use x2APIC, at least if it's a Linux guest. E.g. turn x2APIC on in BIOS, which is often (always?) controlled by the host, and Linux will use x2APIC. In other words, given that AVIC is well on its way to becoming a "legacy" feature, IMO there needs to be a fairly strong use case to justify taking on this much code and complexity. ~1500 lines of code to support a feature that has historically been buggy _without_ nested support is going to require a non-trivial amount of effort to review, stabilize, and maintain. [*] 1235 "Guest With AVIC (Advanced Virtual Interrupt Controller) Enabled May Fail to Process IPI (Inter-Processor Interrupt) Until Guest Is Re-Scheduled" in https://www.amd.com/system/files/TechDocs/56323-PUB_1.00.pdf