From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id D66F7C636D7 for ; Tue, 21 Feb 2023 21:59:27 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:From:References:Cc:To: Subject:MIME-Version:Date:Message-ID:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=NY5fhpnTrx/lFe1T6WdB0cwekfD2G92LurQZL+7sdYM=; b=1go3XdJfJgDq8Q vGr5NTxj8W1x3/s0eUXnarl4UtZKsm6HlgohQQSzfcD672vhQTHNMnHwO/S3PzAg642FrKbzKsXf6 aJnOB5cpNebdU7E6J4DJ47dIHvH+s0fv6dRHR16E4uWvVxFd1HEKUEgCNTUQApVdBJ3Ux6k+T8Arz 8xhCRr9qg8dwdcSq3vDoxqBAXraT+L0xOsrldyGLtbSCV1JGvyTrNsf2xOCeRgxuVYE2p3FlLJB/G +8YRQXmgwVj0hdS0m8R1z7jQ3i2keGEe1dnmrv29quapZyQEiaarmQWAdDho+3kYf7LUet7vORyuZ kEK+Oljv6y60ksvm7zVQ==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1pUaeQ-009tn2-BH; Tue, 21 Feb 2023 21:58:18 +0000 Received: from esa4.hgst.iphmx.com ([216.71.154.42]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1pUaeL-009tlF-Tk for linux-arm-kernel@lists.infradead.org; Tue, 21 Feb 2023 21:58:16 +0000 DKIM-Signature: v=1; a=rsa-sha256; c=simple/simple; d=wdc.com; i=@wdc.com; q=dns/txt; s=dkim.wdc.com; t=1677016693; x=1708552693; h=message-id:date:mime-version:subject:to:cc:references: from:in-reply-to:content-transfer-encoding; bh=g9TnFRqAZqcDsGZcV1iYV9pTjzWi1/1bs6SxLfEGNWA=; b=NU0suVy1uPPghJ3MZOTc+ai2LOSgt8O3xghnIRyTSFSAZ5gbt4fqGotg sGTyV3Ix8zwx5BxWudgxwDmo3XkLQ4XwDzXqA8wK7+hEqsiB/Nwsdo5ys Sy85FlQ8idmkkoLdFhnlt9HHmiV4slcx5qZ4BblVkdOK5wOUpLQDKi8fk HuutYTRFk3Se3SCiO/JRwPTo9RbmWeHEz/NSLEub+EELfnGeH/OEfkw90 sq6/CDGAwQqghkkwkrKhpITn0nj/Dr3HTRSGI/LJY+3DWKNU0TFhgbhq1 u2wudLQNf0dxlDySUMYIozDLrim0hqVfhC51OrpcjzBwn9vpkUh1f8Z0B g==; X-IronPort-AV: E=Sophos;i="5.97,317,1669046400"; d="scan'208";a="222134146" Received: from h199-255-45-14.hgst.com (HELO uls-op-cesaep01.wdc.com) ([199.255.45.14]) by ob1.hgst.iphmx.com with ESMTP; 22 Feb 2023 05:58:05 +0800 IronPort-SDR: o36Aha8T6bPzw8YitgtaSNW26/0LUiH15TvNF0zUM/bHYb47f9ceFfkKjStpzwjfFDGXYVXMGD OsuMRT0YjWs3fH9baU1t5Hp+WEu4RHXJkvdIKoAH8ASf8AVUBM877gyaU1GLcn+Wv41rUH5OgM 5QgprVbf2+Mki6/X7shUrXKpZwnJoDI6xwlhdH6c/e0Z6VMduiLuxAfIWgFVOxvFR4l8WVYBDC iKf46fDDayoT9BAwDJIUVZxg8eqwFcEC0QZldkKNvGPqU/CX54xn4Lwal0D1F9Z4KZT5Toat16 QVQ= Received: from uls-op-cesaip02.wdc.com ([10.248.3.37]) by uls-op-cesaep01.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 21 Feb 2023 13:15:04 -0800 IronPort-SDR: 5qVI7WqwnrVlOtI5IgyU+UCPBCou3LDNzSSyTwql0tzwi+ltxA2yLI5zvS1JMDgW2GX7zT/AcD 778sFjoQiqrUa51O1FvagMSrsqMEcrmZpXPOpcGNAbXwCnmBsEPisDmi4F5S0CoZZezT3FJTZi HSjkL75pjZ5D9l4gk493D1uA0L8jmqvS0CCVDw9ugcw3Lex2JlS8a69i4oy6/Tuhao+83dfS5S FZubwxcOL027+vLlYKScSCsNylybkHiQbY3hnfSWL1SvUEG1NmnSQlT7VSpFznqwfdjteVD3sa 6Vw= WDCIronportException: Internal Received: from usg-ed-osssrv.wdc.com ([10.3.10.180]) by uls-op-cesaip02.wdc.com with ESMTP/TLS/ECDHE-RSA-AES128-GCM-SHA256; 21 Feb 2023 13:58:05 -0800 Received: from usg-ed-osssrv.wdc.com (usg-ed-osssrv.wdc.com [127.0.0.1]) by usg-ed-osssrv.wdc.com (Postfix) with ESMTP id 4PLtSr4tbpz1Rwtl for ; Tue, 21 Feb 2023 13:58:04 -0800 (PST) Authentication-Results: usg-ed-osssrv.wdc.com (amavisd-new); dkim=pass reason="pass (just generated, assumed good)" header.d=opensource.wdc.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d= opensource.wdc.com; h=content-transfer-encoding:content-type :in-reply-to:organization:from:content-language:references:to :subject:user-agent:mime-version:date:message-id; s=dkim; t= 1677016682; x=1679608683; bh=g9TnFRqAZqcDsGZcV1iYV9pTjzWi1/1bs6S xLfEGNWA=; b=Rb9/RTEigoXlpl3/WTaPyfgRujxS0dNLPQlHHKqTpzdykTvexkm mK+7rSBYPQchCE4X3iRubIdTVA1kuy2s3/s3FL3IymSUduI/0C8ClvciOyDM8gqy nThhdHRXBkRNWw68JXTvoGQluphxoEZ+yvmpAAIMCjrncML4W8JH3FV3E3nQyyDG j/xPAI2U+cEZqSkvYKF1xEXUxe0bhttNRB1cf4Vb6tlHY++EsL3TP/kw1vlBMCnB emJm1kL5Pd/hvTIfCk01L9mRDeWcf9IBkZLlW/d4hgwTSMDu8fM8TVnrkkUyQ2zL cfzPhfLndTkVUweNAu5WFR+P0WGYENfno1g== X-Virus-Scanned: amavisd-new at usg-ed-osssrv.wdc.com Received: from usg-ed-osssrv.wdc.com ([127.0.0.1]) by usg-ed-osssrv.wdc.com (usg-ed-osssrv.wdc.com [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id pMLelR1JIQ8e for ; Tue, 21 Feb 2023 13:58:02 -0800 (PST) Received: from [10.225.163.21] (unknown [10.225.163.21]) by usg-ed-osssrv.wdc.com (Postfix) with ESMTPSA id 4PLtSk5wLTz1RvLy; Tue, 21 Feb 2023 13:57:58 -0800 (PST) Message-ID: Date: Wed, 22 Feb 2023 06:57:57 +0900 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.8.0 Subject: Re: [PATCH v2 9/9] PCI: rockchip: Add parameter check for RK3399 PCIe endpoint core set_msi() To: Rick Wertenbroek Cc: alberto.dassatti@heig-vd.ch, xxm@rock-chips.com, rick.wertenbroek@heig-vd.ch, Rob Herring , Krzysztof Kozlowski , Heiko Stuebner , Shawn Lin , Lorenzo Pieralisi , =?UTF-8?Q?Krzysztof_Wilczy=c5=84ski?= , Bjorn Helgaas , Jani Nikula , Greg Kroah-Hartman , Rodrigo Vivi , Mikko Kovanen , devicetree@vger.kernel.org, linux-arm-kernel@lists.infradead.org, linux-rockchip@lists.infradead.org, linux-kernel@vger.kernel.org, linux-pci@vger.kernel.org References: <20230214140858.1133292-1-rick.wertenbroek@gmail.com> <20230214140858.1133292-10-rick.wertenbroek@gmail.com> <38ae72c9-0f0b-1a94-d2e0-f4ea80e94705@opensource.wdc.com> Content-Language: en-US From: Damien Le Moal Organization: Western Digital Research In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20230221_135814_158234_665F566D X-CRM114-Status: GOOD ( 51.27 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On 2/21/23 22:19, Rick Wertenbroek wrote: > On Tue, Feb 21, 2023 at 11:55 AM Damien Le Moal > wrote: >> >> On 2/21/23 19:47, Rick Wertenbroek wrote: >>> On Wed, Feb 15, 2023 at 2:39 AM Damien Le Moal >>> wrote: >>>> >>>> On 2/14/23 23:08, Rick Wertenbroek wrote: >>>>> The RK3399 PCIe endpoint core supports only a single PCIe physcial >>>>> function (function number 0), therefore return -EINVAL if set_msi() is >>>>> called with a function number greater than 0. >>>>> The PCIe standard only allows the multi message capability (MMC) value >>>>> to be up to 0x5 (32 messages), therefore return -EINVAL if set_msi() is >>>>> called with a MMC value of over 0x5. >>>>> >>>>> Signed-off-by: Rick Wertenbroek >>>>> --- >>>>> drivers/pci/controller/pcie-rockchip-ep.c | 10 ++++++++++ >>>>> 1 file changed, 10 insertions(+) >>>>> >>>>> diff --git a/drivers/pci/controller/pcie-rockchip-ep.c b/drivers/pci/controller/pcie-rockchip-ep.c >>>>> index b7865a94e..80634b690 100644 >>>>> --- a/drivers/pci/controller/pcie-rockchip-ep.c >>>>> +++ b/drivers/pci/controller/pcie-rockchip-ep.c >>>>> @@ -294,6 +294,16 @@ static int rockchip_pcie_ep_set_msi(struct pci_epc *epc, u8 fn, u8 vfn, >>>>> struct rockchip_pcie *rockchip = &ep->rockchip; >>>>> u32 flags; >>>>> >>>>> + if (fn) { >>>>> + dev_err(&epc->dev, "This endpoint controller only supports a single physical function\n"); >>>>> + return -EINVAL; >>>>> + } >>>> >>>> Checking this here is late... Given that at most only one physical >>>> function is supported, the check should be in rockchip_pcie_parse_ep_dt(). >>>> Something like: >>>> >>>> err = of_property_read_u8(dev->of_node, "max-functions", >>>> &ep->epc->max_functions); >>>> >>>> if (err < 0 || ep->epc->max_functions > 1) >>>> >>>> ep->epc->max_functions = 1; >>>> >>> >>> Yes, this could be moved to the probe, thanks. >>> >>>> And all the macros with the (fn) argument could also be simplified >>>> (argument fn removed) since fn will always be 0. >>> >>> These functions cannot be simplified because they have to follow the signature >>> given by "pci_epc_ops" (include/linux/pci-epc.h). And this signature has the >>> function number as a parameter. If we change the function signature we won't >>> be able to assign these functions to the pc_epc_ops structure >> >> I was not suggesting to change the functions signature. I was suggesting >> dropping the fn argument for the *macros*, e.g. >> >> ROCKCHIP_PCIE_EP_FUNC_BASE(fn) -> ROCKCHIP_PCIE_EP_FUNC_BASE >> >> since fn is always 0. >> >> That said, I am not entirely sure if the limit really is 1 function at most. The >> TRM seems to be suggesting that up to 4 functions can be supported... >> >> [...] >> >>>> Another nice cleanup: define ROCKCHIP_PCIE_EP_MSI_CTRL_REG to include the >>>> ROCKCHIP_PCIE_EP_FUNC_BASE(fn) addition so that we do not have to do it >>>> here all the time. >>> >>> Yes, this could be an improvement but this is the way it is written >>> everywhere in this >>> driver, I chose to keep it so as to remain coherent with the rest of the driver. >>> Cleaning this is not so important since this code will not be >>> rewritten / changed so >>> often. But I agree that it might be nicer. But, on the other side if >>> at some point >>> support for virtual functions would be added, the offsets would need >>> to be computed >>> based on the virtual function number and the code would be written >>> like it is now, >>> so I suggest keeping this the way it is for now. >> >> Yes, sure, this can be cleaned later. >> >> A more pressing problem is the lack of support for MSIX despite the fact that >> the controller supports that *and* advertize it as a capability. That is what >> was causing my problem with the Linux nvme driver and my prototype nvme epf >> function driver: the host driver was seeing MSIX support (1 vector supported by >> default), and so was allocating one MSIX for the device probe. But on the EP >> end, it is MSI or INTX only... Working on adding that to solve this issue. >> > > I have seen this too, the controller advertises the capability. However, the TRM > (section 17.5.9) says that MSI-X is not supported (MSI / INTx only as you said). > So the solution should be to modify the probe function of the endpoint > controller > so that the MSI-X capability would not be advertised, this should fix > your problem. Yep, that is what I did for now: write 0 in the capability ID of the MSIX capability list entry. A cleaner solution would be to change the next entry pointer of the entry preceding the MSIX entry. Will send a patch for that. > > I wonder if one could still implement MSI-X because from the endpoint we would > just need to implement it as a message (TLP) over PCIe (because the space for > the vectors is allocated and written, so that part should be ok). I am > not an expert > on MSI-X, but the reason the endpoint cannot send them could be because MSI-X > requires some fields in the PCIe header descriptor to be filled with values that > cannot be set through the "desc0-3" registers of the RK3399 PCIe endpoint core. > > Anyways, the endpoint should not advertise the MSI-X capabilities when it cannot > send such interrupts. Once this is fixed you should be able to have your NVMe > function running. > > Regards. > Rick > > >> -- >> Damien Le Moal >> Western Digital Research >> -- Damien Le Moal Western Digital Research _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel