From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,MAILING_LIST_MULTI, NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DC6AC433DB for ; Tue, 9 Feb 2021 11:01:45 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by mail.kernel.org (Postfix) with ESMTP id A656764E50 for ; Tue, 9 Feb 2021 11:01:44 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232212AbhBILBO (ORCPT ); Tue, 9 Feb 2021 06:01:14 -0500 Received: from szxga01-in.huawei.com ([45.249.212.187]:4617 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S232006AbhBIK6e (ORCPT ); Tue, 9 Feb 2021 05:58:34 -0500 Received: from dggeme706-chm.china.huawei.com (unknown [172.30.72.55]) by szxga01-in.huawei.com (SkyGuard) with ESMTP id 4DZfvg055MzY78B; Tue, 9 Feb 2021 18:56:15 +0800 (CST) Received: from [10.174.63.109] (10.174.63.109) by dggeme706-chm.china.huawei.com (10.1.199.102) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.2106.2; Tue, 9 Feb 2021 18:57:29 +0800 Subject: Re: [v3] PCI: Add pci reset quirk for Huawei Intelligent NIC virtual function To: Bjorn Helgaas , Alex Williamson CC: "bhelgaas@google.com" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "Yinshi (Stone)" , "Wangxiaoyun (Cloud)" , zengweiliang zengweiliang , "Chenlizhong (IT Chip)" References: <20210121153043.GA2654954@bjorn-Precision-5520> From: Chiqijun Message-ID: <63b84959-d649-486a-3736-bbfca46c2362@huawei.com> Date: Tue, 9 Feb 2021 18:57:29 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:68.0) Gecko/20100101 Thunderbird/68.1.1 MIME-Version: 1.0 In-Reply-To: <20210121153043.GA2654954@bjorn-Precision-5520> Content-Type: text/plain; charset="utf-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [10.174.63.109] X-ClientProxiedBy: dggeme718-chm.china.huawei.com (10.1.199.114) To dggeme706-chm.china.huawei.com (10.1.199.102) X-CFilter-Loop: Reflected Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org On 2021/1/21 23:30, Bjorn Helgaas wrote: > [Alex is a reset expert, hoping he can chime in] > > On Thu, Jan 21, 2021 at 08:53:12PM +0800, Chiqijun wrote: >> On 2021/1/9 6:25, Bjorn Helgaas wrote: >>> On Fri, Dec 25, 2020 at 05:25:30PM +0800, Chiqijun wrote: >>>> When multiple VFs do FLR at the same time, the firmware is >>>> processed serially, resulting in some VF FLRs being delayed more >>>> than 100ms, when the virtual machine restarts and the device >>>> driver is loaded, the firmware is doing the corresponding VF >>>> FLR, causing the driver to fail to load. >>>> >>>> To solve this problem, add host and firmware status synchronization >>>> during FLR. >>>> >>>> Signed-off-by: Chiqijun >>>> ... > >>>> + * Get and check firmware capabilities. >>>> + */ >>>> + val = readl(bar + HINIC_VF_FLR_TYPE); >>>> + if (!(val & (1UL << HINIC_VF_FLR_CAP_BIT_SHIFT))) { >>>> + pci_iounmap(pdev, bar); >>>> + return -ENOTTY; >>>> + } >>>> + >>>> + /* >>>> + * Set the processing bit for the start of FLR, which will be cleared >>>> + * by the firmware after FLR is completed. >>>> + */ >>>> + val = readl(bar + HINIC_VF_OP); >>>> + val = val | (1UL << HINIC_VF_FLR_PROC_BIT_SHIFT); >>>> + writel(val, bar + HINIC_VF_OP); >>>> + >>>> + /* Perform the actual device function reset */ >>>> + pcie_flr(pdev); >>>> + >>>> + /* >>>> + * The device must learn BDF after FLR in order to respond to BAR's >>>> + * read request, therefore, we issue a configure write request to let >>>> + * the device capture BDF. >>>> + */ >>>> + pci_read_config_word(pdev, PCI_COMMAND, &command); >>>> + pci_write_config_word(pdev, PCI_COMMAND, command); >>> >>> I assume this is because of this requirement from PCIe r5.0, sec >>> 2.2.9: >>> >>> Functions must capture the Bus and Device Numbers supplied with all >>> Type 0 Configuration Write Requests completed by the Function, and >>> supply these numbers in the Bus and Device Number fields of the >>> Completer ID for all Completions generated by the Device/Function. >>> >>> I'm a little concerned because it seems like this requirement should >>> apply to *all* resets, and I don't see where we do a similar write >>> following other resets. Can you help me out? Do we need this in >>> other cases? Do we do it? >> >> This depends on the hardware device. The HINIC device clears the BDF >> information of the VF during FLR, so it relies on Configuration >> Write Requests to capture BDF. If other devices do not clear the DBF >> information during FLR, this operation is not required. > > If the spec says devices must keep the latched BDF during FLR, and the > HINIC doesn't comply with that, then it makes sense to do a config > write here in HINIC-specific code. > > But if devices are allowed to clear the BDF during FLR, the OS has to > assume they all do, and the generic code for FLR (and probably other > resets) should do a config write so devices can latch the BDF again. > >> In addition, I did not find other devices directly access the BAR register >> after FLR in resets. > > I didn't catch your meaning here. > > If a device loses the BDF during FLR and we don't do something to > allow it to latch the BDF again, any completions from the device will > have the wrong information. We will likely do *some* config write to > the device eventually, which will fix this, but we can't rely on some > unknown future write to do this. If it's a problem, we need to > explicitly do a write for this purpose. > > Bjorn > . > > The spec does not specify whether the BDF needs to be kept after FLR, but the section 2.2.9 of the PCIe r5.0 has the following description: If a Function must generate a Completion prior to the initial device Configuration Write Request, 0's must be entered into the Bus Number and Device Number fields Does this mean that we should always get the expected completion after initializing the device?