From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1751965AbcLFGHh (ORCPT ); Tue, 6 Dec 2016 01:07:37 -0500 Received: from cn.fujitsu.com ([59.151.112.132]:48558 "EHLO heian.cn.fujitsu.com" rhost-flags-OK-FAIL-OK-FAIL) by vger.kernel.org with ESMTP id S1751680AbcLFGHg (ORCPT ); Tue, 6 Dec 2016 01:07:36 -0500 X-IronPort-AV: E=Sophos;i="5.22,518,1449504000"; d="scan'208";a="13641948" Subject: Re: [PATCH] vfio/pci: Support error recovery To: Alex Williamson References: <1480246457-10368-1-git-send-email-caoj.fnst@cn.fujitsu.com> <20161130210413.5161aab1@t450s.home> <58402830.3060606@cn.fujitsu.com> <20161201075541.756f6332@t450s.home> <5844092A.30204@cn.fujitsu.com> <20161204083047.7e715b09@t450s.home> <58450083.9010201@cn.fujitsu.com> <20161205091730.568e5079@t450s.home> CC: , , , From: Cao jin Message-ID: <58465677.1000403@cn.fujitsu.com> Date: Tue, 6 Dec 2016 14:11:03 +0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:38.0) Gecko/20100101 Thunderbird/38.1.0 MIME-Version: 1.0 In-Reply-To: <20161205091730.568e5079@t450s.home> Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.167.226.69] X-yoursite-MailScanner-ID: 681E2467048F.A4673 X-yoursite-MailScanner: Found to be clean X-yoursite-MailScanner-From: caoj.fnst@cn.fujitsu.com Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 12/06/2016 12:17 AM, Alex Williamson wrote: > On Mon, 5 Dec 2016 13:52:03 +0800 > Cao jin wrote: > >> On 12/04/2016 11:30 PM, Alex Williamson wrote: >>> On Sun, 4 Dec 2016 20:16:42 +0800 >>> Cao jin wrote: >>> >>>> On 12/01/2016 10:55 PM, Alex Williamson wrote: >>>>> On Thu, 1 Dec 2016 21:40:00 +0800 >>>> >>>>>>> If an AER fault occurs and the user doesn't do a reset, what >>>>>>> happens when that device is released and a host driver tries to make >>>>>>> use of it? The user makes no commitment to do a reset and there are >>>>>>> only limited configurations where we even allow the user to perform a >>>>>>> reset. >>>>>>> >>>>>> >>>>>> Limited? Do you mean the things __pci_dev_reset() can do? >>>>> >>>>> I mean that there are significant device and guest configuration >>>>> restrictions in order to support AER. For instance, all the functions >>>>> of the slot need to appear in a PCI-e topology in the guest with all >>>>> the functions in the right place such that a guest bus reset translates >>>>> into a host bus reset. The physical functions cannot be split between >>>>> guests even if IOMMU isolation would otherwise allow it. The user >>>>> needs to explicitly enable AER support for the devices. A VM need to >>>>> be specifically configured for AER support in order to set any sort of >>>>> expectations of a guest directed bus reset, let alone a guarantee that >>>>> it will happen. So all the existing VMs, where functions are split >>>>> between guests, or the topology isn't exactly right, or AER isn't >>>>> enabled see a regression from the above change as the device is no >>>>> longer reset. >>>>> >>>> >>>> I am not clear why set these restrictions in the current design. I take >>>> a glance at older versions of qemu's patchset, their thoughts is: >>>> translate a guest bus reset into a host bus reset(Which is >>>> unreasonable[*] to me). And I guess, that's the *cause* of these >>>> restrictions? Is there any other stories behind these restrictions? >>>> >>>> [*] In physical world, set bridge's secondary bus reset would send >>>> hot-reset TLP to all functions below, trigger every device's reset >>>> separately. Emulated device should behave the same, means just using >>>> each device's DeviceClass->reset method. >>> >>> Are you trying to say that an FLR is equivalent to a link reset? >> >> No. Look at old versions patchset, there is one names "vote the >> function 0 to do host bus reset when aer occurred"[1], that is what I >> called "translate guest link reset to host link reset", and what I think >> unreasonable(and I think it also does it wrongly). So in v10 version of >> mine, I dropped it. >> >> [1]https://lists.gnu.org/archive/html/qemu-devel/2016-05/msg02987.html >> >> If "translate guest link reset to host link reset" is right, I can >> understand these restrictions[2][3]. >> >> [2]. All physical functions in a single card must be assigned to the VM >> with AER enabled on each and configured on the same virtual bus. >> [3]. Don't place other devices under the virtual bus in [2], no matter >> physical, emulated, or paravirtual, even if other device >> supporting AER signaling >> >> Certain device's FLR calls its DeviceClass->reset method; link reset >> calls DeviceClass->reset of each device which on the bus. So, apparently >> they have difference. But if there is only 1 vfio-pci device under the >> virtual pci bus, I think FLR can be equivalent to a link reset, right? > > No. An FLR resets the device while a secondary bus reset does a reset > of the link and the device. AER errors are sometimes issues with the > link, not the device. If we were to perform only an FLR, we're not > performing the same reset as would be done on bare metal. > Thanks for you explanation, it does helps, except the last sentence, I think I understand it now: fatal error implies there may be link issue exists(pci express spec: 6.2.2.2.1), so, should do link reset for fatal error(that is what and why aer core does). And so, in patch[1] above, qemu does a link reset when seeing secondary bus reset bit of virtual bus got set. is it right? -- Sincerely, Cao jin