From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id BAF72C77B61 for ; Tue, 11 Apr 2023 02:37:28 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:References: Message-ID:Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=/MdckM1w5O5G3nN1C05tveKLfU1L0uem5lsNFtXEvJs=; b=f7jgKg4tfcJFhp 4nZCiSZnof+9+ra7Rx9LblrdUW7vX0sCkKcDrArSndblG4gtiK9olo6pIwTHEo6ssBv6sqgj5Lx/q RVBWm9M3u4C/DjaJEpvMEYpp42kngfoMIZHliR3Hno4S3c/C8QFvhwMD+2j/1xN340+yEKSIokuf/ tKsHgokqoWN2v7/1N9NucHUlprtBISwI7gdbxn6L7ZzwCoc0pmhvN0ts1cO+xxxKhbxfRLQu2OWSv PJGLJbGK32CHf4dmX2W082ZYe1IYcIklDXzQ4GWwEf38Sp74ad7hR5vHebO2WfWscLjiQj28VytCq GWz01Jv8MlPUzq7uGaqA==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.96 #2 (Red Hat Linux)) id 1pm3s8-00GGuF-2y; Tue, 11 Apr 2023 02:36:40 +0000 Received: from mail-lf1-x12c.google.com ([2a00:1450:4864:20::12c]) by bombadil.infradead.org with esmtps (Exim 4.96 #2 (Red Hat Linux)) id 1pm3s5-00GGth-0c for linux-arm-kernel@lists.infradead.org; Tue, 11 Apr 2023 02:36:38 +0000 Received: by mail-lf1-x12c.google.com with SMTP id z8so10428211lfb.12 for ; Mon, 10 Apr 2023 19:36:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; t=1681180594; x=1683772594; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=ryXtbfRI/jstwh7DoB7ytVR8A4UxPsGFr890Bq2wcsw=; b=Fs5DzYjPDnocAKCDNiD3JWo3tpGPaB3GrE9UZJokrnPBHpaBE/07CH2AXiTOaJ/OQS 7NOQJvU1xuuZ9bzqbVyy6Pv9NdV76x+Fw/rAI/2zlSjeTtUJK4RWJWdibP+9kLbR0T04 iMxP4WoV7Ar4HjbHrVcIP+DalirmK4amqrHdEjT7dP8RF02PVLhcRDlBP1tegOgmNYuw KfahlM37LQTuFX8TXxaCVdC9C+hMqFlz5E+GCqOK9jUo0OoWEK4bnKVkVYsMr2OhmUp/ 0SWO8yaU9driTfWmAraoXak4vl2vLNIi8s0YjX4ToOecI7DMya+e8U0EL1wv6gSAQHiG 8OjA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; t=1681180594; x=1683772594; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=ryXtbfRI/jstwh7DoB7ytVR8A4UxPsGFr890Bq2wcsw=; b=w7lCc1jqsHhHpxtuQdFM864Rei00n1/EII9ikdX0MxK6c6HGNyZ3TKFWUZXCIvvary boTDMxJ7SoPLCIl2S+ELS9+TwU3xwXwagzZeVVeM5Umsz/fx1p7Arztpq41ZBRuofanR IoreyeaOD0TWS19CQq5cw9laU0NFOvDGdccV4uMc636NOYVcXfqRa0QKyLGF6P7+A7P/ hvDgMB0F9t+shEwG3fqStTdK7axYM3pt3J1/pHWZyD4+galmLnfwebAOqHFtsV9jMywf LuXDqkiaTS73oLZArlAh3+/4advu4E28YgMhiGF0HS9F/X3gNowKXEql7JwshY7cRzsp Q0wA== X-Gm-Message-State: AAQBX9dAQ1aAaR2FwsFbAB3dmSWJkruw7qppWfq8THusHFY9OIUeExJ2 S8284/CYJ4B6xw2VyKfQvdmvYZMJiHg= X-Google-Smtp-Source: AKy350bsFHTiFElDI8LjkPIhUS1yq/z4Zcc/dQLQe3pSWHfUqlWAPWjMVTzYlehrO1zaO/ZhcCtaRw== X-Received: by 2002:a19:f617:0:b0:4eb:982:adf with SMTP id x23-20020a19f617000000b004eb09820adfmr2942231lfe.26.1681180593693; Mon, 10 Apr 2023 19:36:33 -0700 (PDT) Received: from mobilestation ([95.79.140.35]) by smtp.gmail.com with ESMTPSA id b7-20020ac24107000000b004e9b307d2c8sm2331899lfi.238.2023.04.10.19.36.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 10 Apr 2023 19:36:33 -0700 (PDT) Date: Tue, 11 Apr 2023 05:36:31 +0300 From: Serge Semin To: Lorenzo Pieralisi Cc: Hongxing Zhu , Bjorn Helgaas , "l.stach@pengutronix.de" , "bhelgaas@google.com" , "linux-pci@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , "linux-kernel@vger.kernel.org" , "kernel@pengutronix.de" , dl-linux-imx , Serge Semin Subject: Re: [PATCH v2] PCI: imx6: Save and restore MSI control of RC in suspend and resume Message-ID: <20230411023631.3g7oom3mj6sp27k4@mobilestation> References: <20230317222436.GA1978818@bhelgaas> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230410_193637_271727_869A8BEE X-CRM114-Status: GOOD ( 38.19 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Wed, Apr 05, 2023 at 05:55:48PM +0200, Lorenzo Pieralisi wrote: > [+Cc Sergey] > Hi Lorenzo [nip] > > > > > > That's because a PCI root complex is not a PCI device (and this is not an MSI > > > capability, which lives in PCI config space). > > > > > > I will reword it (and the commit log with it) and merge it in the coming weeks for > > > v6.4 > > Hi Lorenzo: > > Thanks a lot for your kindly help. > > I am getting back to this since I am still not convinced and I want to > understand this once for all. > > We do use dw_pcie_find_capability() in most DWC drivers to find and > peek/poke at eg PCI express capability of the *Root port* (?), > > eg dw_pcie_wait_for_link() > > so I assume that for iMX6 dw_pcie_find_capability() does just the same, > which would mean that we are poking the "Message Control" field of the > Root port MSI capability. Absolutely right. Besides of the standard PCIe config space which is filled with the various PCIe capability registers, the DW PCIe Root Port has the "Port Logic" CSR space with the DW PCIe-specific settings: GENx Link control and lanes tuning stuff, iMSI-Rx engine, iATU/eDMA, etc. Some of the capabilities are utilized by their direct functionality (like the PCI Express capability) meanwhile some are left unused even though they can be activated during the IP-core synthesize stage. I don't know why the HW designers left them being optionally enabled and then available for some setups, DW PCIe RC HW-manual doesn't have their functionality description. The MSI capability is one of such PCIe capabilities. It's described in the DW PCIe RC registers space HW-manual and the DW PCIe RC IP-core parameters, but the rest of the manual text has nothing about it. Also note even though the DW PCIe RC HW-manual has _RC_ in the name it actually describes the PCIe Root Port with the attention remark: "The Synopsys PCI Express IP does not implement a full root complex." So from what I can see there can be several reasons of such configuration: 1. It simplifies the dual-mode device design for the HW-engineers. Though IMO Synopsys at least should have dropped the unused capabilities from the RP HW-manual. 2. Synopsys may imply that these capabilities can be used somehow for the vendor-specific true Root Complex designs. Though they don't specifically say about that. Instead there is a note: "To implement a full root complex, you must add your application logic (connected to the application client interface or AXI interface) and other support logic (including clock and reset generation)." > > Either that (which would mean that iMX6 has a HW bug because the RP > Message Control field does not control the delivery of MSIs from > endpoints but just for the root port itself ) I wouldn't call it a HW bug seeing the MSI capability registers are actually unused by the RP controllers. So it looks more like the iMX6-specific feature. See a short explanation in the commit 75cb8d20c112 ("PCI: imx: Enable MSI from downstream components"). Normally MSI event delivery to the CPU doesn't depend on the actual MSI capability CSRs state in DW PCIe RPs. Instead the MSI is configured via the Port Logic iMSI-RX registers. That's what is done in the DW PCIe host driver (see pcie-designware-host.c). > or all DWC controllers > modelled the root complex MMIO space as a set of PCI/PCIe capabilities > that are NOT necessarily mappable to PCI specifications defined ones. Basically you are right though as I noted above Synopsys doesn't provide a complete RC solution. It's just a single RP with the MMIO (DBI) CSRs looking as the PCIe TYPE1 config space. -Serge(y) > > Can anyone please shed some light on this ? I don't have DWC HW, we need > to know before merging this code. > > Thanks, > Lorenzo > > > > > Best Regards > > Richard Zhu > > > > > > Thanks, > > > Lorenzo > > > > > > > + * > > > > + * The MSI_EN bit of MSI control register contained in this MSI-CAP > > > > + * is used control the MSI delivery of MSI interrupts from components > > > > + * below the Root Port. > > > > + * > > > > + * Find it by PCI_CAP_ID_MSI here, and assert the MSI_EN bit to > > > allow > > > > + * the MSI delivery below the Root Port, if the PCI MSI is enabled. > > > > + */ > > > > if (pci_msi_enabled()) { > > > > offset = dw_pcie_find_capability(pci, PCI_CAP_ID_MSI); > > > > dw_pcie_dbi_ro_wr_en(pci); Best Regards Richard Zhu > > > > > > > > > > Bjorn > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel