From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1671DC28B2B for ; Fri, 19 Aug 2022 08:38:02 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S230077AbiHSIh7 (ORCPT ); Fri, 19 Aug 2022 04:37:59 -0400 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:53820 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1345812AbiHSIhS (ORCPT ); Fri, 19 Aug 2022 04:37:18 -0400 Received: from mail-pj1-x1031.google.com (mail-pj1-x1031.google.com [IPv6:2607:f8b0:4864:20::1031]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 32288EEF08 for ; Fri, 19 Aug 2022 01:35:52 -0700 (PDT) Received: by mail-pj1-x1031.google.com with SMTP id f21so3998393pjt.2 for ; Fri, 19 Aug 2022 01:35:52 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linaro.org; s=google; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date:from:to :cc; bh=QqmEKIFzHAXTFm3nLmoojS/pMmtd2oRkzT9dCGl8OZA=; b=KQ3U7FTXxenS+f3Dlniw65UAFkZujmauHkuWxWyD153F65mpDvn8msgcjRm/5yK3gK TXF9xexHG28nUlb5RyVjsLOKxIogCCbJOccKvEuDZIcHgXVCwFrSx72tDBIDEENLVjmx hIcL72VUa7PuMZdsKoB5EJLNh9Zm92WfPqk08WLLIMsFSDVl6NdgDNgp+KCcjqt/Lx1e EkMi+hhAKXwGcvc2O822gJBHYleidNaaW5ErYfPuSRw8KHy25JUXTlytRhG7Zadbwew6 pKdKUX8RRVeROm6p7CmV0L4RYIVha0+dWU5ZIuJgZzAyU/IuQDhyJO2K9SNeiBYNwPmT dEdA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=in-reply-to:content-transfer-encoding:content-disposition :mime-version:references:message-id:subject:cc:to:from:date :x-gm-message-state:from:to:cc; bh=QqmEKIFzHAXTFm3nLmoojS/pMmtd2oRkzT9dCGl8OZA=; b=x3ectgGYrajX709hr7seJHoCFzbFt2LjUtYznLlqNCdmPc/Hze+3IU4Sk/QqQTR6tW G3BkbJUWymyTOx58Bv6ozUMCkgOcXPZWi1p/SiGQaVtzZWIyZoMEt6O82EJDBBraHi22 ruyUdso8Q1b5zyFI7YyzwG7Cd96KOAEKozijCQw4LCunqq9ZRe+riX2qu6XM2kPtZofH swvjaFZ+36MEKOgdeVq4NjQ8dyugSRJcjGT7L+kYPfgDWhphlY8A00tqLQNur/i4Luzs MHXMMRQeUTXJXu0hPv5z+hZNtNiykNwcVu6rwtZj2Gtb3PlTRRDWtHEVz0UKN0d8I0A+ C+Jw== X-Gm-Message-State: ACgBeo1nXcgFxhcwGSwJ/edL9nnqvg/vWki42eh3aT/RnGhLtRXez+Bz 1LEKl8hjo+sRbrPLFGXt6CQX X-Google-Smtp-Source: AA6agR77D90sJTcBkFusO14uw8jiPP6rbN3f/CGnflS/R3U6NqQHXp1Xg4HkMnMXbQS7M8oLS71giA== X-Received: by 2002:a17:90a:318a:b0:1fa:c68c:aa03 with SMTP id j10-20020a17090a318a00b001fac68caa03mr7774546pjb.172.1660898151402; Fri, 19 Aug 2022 01:35:51 -0700 (PDT) Received: from thinkpad ([117.193.212.74]) by smtp.gmail.com with ESMTPSA id d135-20020a621d8d000000b0052d4b0d0c74sm2949697pfd.70.2022.08.19.01.35.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 19 Aug 2022 01:35:50 -0700 (PDT) Date: Fri, 19 Aug 2022 14:05:41 +0530 From: Manivannan Sadhasivam To: Vidya Sagar Cc: Lorenzo Pieralisi , Rob Herring , Bjorn Helgaas , Jingoo Han , Gustavo Pimentel , Krzysztof Wilczynski , Bjorn Helgaas , PCI , "linux-kernel@vger.kernel.org" , Krishna Thota , Manikanta Maddireddy , sagar.tv@gmail.com, Xiaowei Bao , Hou Zhiqiang Subject: Re: [PATCH V1] PCI: designware-ep: Fix DBI access before core init Message-ID: <20220819083541.GB215264@thinkpad> References: <051a3baf-b4dd-7764-2e61-03584cefb4d3@nvidia.com> <20220729224404.GA478920@bhelgaas> <20220730145025.GA4005@thinkpad> <20220802072426.GA2494@thinkpad> <20220802140738.GA115652@thinkpad> <5d9f537b-a7a3-b8ae-8779-9ffc15fd11bb@nvidia.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <5d9f537b-a7a3-b8ae-8779-9ffc15fd11bb@nvidia.com> Precedence: bulk List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Tue, Aug 16, 2022 at 08:05:08PM +0530, Vidya Sagar wrote: > > > On 8/16/2022 7:45 PM, Lorenzo Pieralisi wrote: > > External email: Use caution opening links or attachments > > > > > > On Tue, Aug 02, 2022 at 07:37:38PM +0530, Manivannan Sadhasivam wrote: > > > On Tue, Aug 02, 2022 at 12:54:37PM +0530, Manivannan Sadhasivam wrote: > > > > On Mon, Aug 01, 2022 at 02:27:14PM -0600, Rob Herring wrote: > > > > > On Sat, Jul 30, 2022 at 8:50 AM Manivannan Sadhasivam > > > > > wrote: > > > > > > > > > > > > On Fri, Jul 29, 2022 at 05:44:04PM -0500, Bjorn Helgaas wrote: > > > > > > > [+cc Xiaowei (author of 6bfc9c3a2c70), Hou (author of 8bcca2658558)] > > > > > > > > > > > > > > On Thu, Jul 28, 2022 at 05:56:28PM +0530, Vidya Sagar wrote: > > > > > > > > On 7/28/2022 3:44 AM, Bjorn Helgaas wrote: > > > > > > > > > On Wed, Jun 22, 2022 at 09:31:33AM +0530, Vidya Sagar wrote: > > > > > > > > > > Platforms that cannot support their core initialization without the > > > > > > > > > > reference clock from the host, implement the feature 'core_init_notifier' > > > > > > > > > > to indicate the DesignWare sub-system about when their core is getting > > > > > > > > > > initialized. Any accesses to the core (Ex:- DBI) would result in system > > > > > > > > > > hang in such systems (Ex:- tegra194). This patch moves any access to the > > > > > > > > > > core to dw_pcie_ep_init_complete() API which is effectively called only > > > > > > > > > > after the core initialization. > > > > > > > > > > > > 6) What's going on with the CORE_INIT and LINK_UP notifiers? > > > > > > > dw_pcie_ep_init_notify() is only called by qcom and tegra. > > > > > > > dw_pcie_ep_linkup() is only called by dra7xx, qcom, and tegra. > > > > > > > As far as I can tell, nobody at all registers to handle those > > > > > > > events except a test. I think it's pointless to have that code > > > > > > > if nobody uses it. > > > > > > > > > > > > > > > > > > > I have submitted an actual driver that makes use of these notifiers: > > > > > > https://lore.kernel.org/lkml/20220502060611.58987-9-manivannan.sadhasivam@linaro.org/ > > > > > > > > > > Notifiers aren't the best interface in the kernel. I think they are > > > > > best used if there's no real linkage between the sender and receiver. > > > > > For an EPC and EPF that's a fixed interface, so define a proper > > > > > interface. > > > > > > > > > > > > > Fair point! The use of notifiers also suffer from an issue where the notifier > > > > chain in EPC is atomic but the EPF calls some of the functions like > > > > pci_epc_write_header() could potentially sleep. > > > > > > > > I'll try to come up with an interface. > > > > > > > > > > I thought about using a new set of callbacks that define the EPC events and > > > have the EPF drivers populate them during probe time. Like below, > > > > > > ``` > > > diff --git a/include/linux/pci-epf.h b/include/linux/pci-epf.h > > > index e03c57129ed5..45247802d6f7 100644 > > > --- a/include/linux/pci-epf.h > > > +++ b/include/linux/pci-epf.h > > > @@ -74,6 +74,20 @@ struct pci_epf_ops { > > > struct config_group *group); > > > }; > > > > > > +/** > > > + * struct pci_epf_events - Callbacks for capturing the EPC events > > > + * @init_complete: Callback for the EPC initialization complete event > > > + * @link_up: Callback for the EPC link up event > > > + */ > > > +struct pci_epc_events { > > > + void (*init_complete)(struct pci_epf *epf); > > > + void (*link_up)(struct pci_epf *epf); > > > +}; > > > + > > > /** > > > * struct pci_epf_driver - represents the PCI EPF driver > > > * @probe: ops to perform when a new EPF device has been bound to the EPF driver > > > @@ -172,6 +186,7 @@ struct pci_epf { > > > unsigned int is_vf; > > > unsigned long vfunction_num_map; > > > struct list_head pci_vepf; > > > + struct pci_epc_events *events; > > > }; > > > > > > /** > > > ``` > > > > > > When each of the event is received by the EPC driver, it will use the EPC API > > > to call the relevant event callback for _each_ EPF. Like below: > > > > > > ``` > > > diff --git a/drivers/pci/endpoint/pci-epc-core.c b/drivers/pci/endpoint/pci-epc-core.c > > > index 6ad9b38b63a9..4b0b30b91403 100644 > > > --- a/drivers/pci/endpoint/pci-epc-core.c > > > +++ b/drivers/pci/endpoint/pci-epc-core.c > > > @@ -724,10 +724,15 @@ EXPORT_SYMBOL_GPL(pci_epc_linkdown); > > > */ > > > void pci_epc_init_notify(struct pci_epc *epc) > > > { > > > + struct pci_epf *epf; > > > + > > > if (!epc || IS_ERR(epc)) > > > return; > > > > > > - blocking_notifier_call_chain(&epc->notifier, CORE_INIT, NULL); > > > + list_for_each_entry(epf, &epc->pci_epf, list) { > > > + if (epf->events->init_complete) > > > + epf->events->init_complete(epf); > > > + } > > > } > > > EXPORT_SYMBOL_GPL(pci_epc_init_notify); > > > ``` > > > > > > Does this look good to you? I can spin up an RFC series, but wanted to check the > > > interface design beforehand. > > > > I am resuming patch reviews, have you posted a follow up ? > > > > Just to understand where we are with this thread and start reviewing > > from there, I will update patchwork accordingly (you should add > > a Link: to this thread anyway in the new series). > > Manivannan posted a new patch set at > https://patchwork.kernel.org/project/linux-pci/list/?series=666818 to > address concerns with the notifier mechanism. > > I would be sending a follow-up patch for the current patch soon. > While posting the new revision, please move dw_pcie_edma_detect() and ep_init() to init_complete function. They also perform DBI access. Thanks, Mani > Thanks, > Vidya Sagar > > > > > Thanks, > > Lorenzo > > > > > Thanks, > > > Mani > > > > > > > Thanks, > > > > Mani > > > > > > > > > Rob > > > > > > > > -- > > > > மணிவண்ணன் சதாசிவம் > > > > > > -- > > > மணிவண்ணன் சதாசிவம் -- மணிவண்ணன் சதாசிவம்