Re: [PATCH v7 1/2] PCI: designware-ep: Fix DBI access before core init

[Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>]

On Thu, Jan 18, 2024 at 06:33:35PM +0000, Niklas Cassel wrote:
> Hello Mani,
> 
> On Sat, Jan 06, 2024 at 08:42:38PM +0530, Manivannan Sadhasivam wrote:
> > > 
> > > So to give you a clear example:
> > > If you:
> > > 1) start the EP, start the pci-epf-test
> > > 2) start the RC
> > > 3) run pci-epf-test
> > > 4) reboot the RC
> > > 
> > > this will trigger a link-down reset IRQ on the EP side (link_req_rst_not).
> > > 
> > 
> > Right. This is the sane RC reboot scenario. Although, there is no guarantee
> > that the EP will get LINK_DOWN event before perst_assert (I've seen this with
> > some RC platforms).
> > 
> > But can you confirm whether your EP is receiving PERST assert/deassert when RC
> > reboots?
> 
> Yes, it does:
> 
> ## RC side:
> # reboot
> 
> ## EP side
> [  845.606810] pci: PERST asserted by host!
> [  845.657058] pci: PCIE_CLIENT_INTR_STATUS_MISC: 0x4
> [  845.657759] pci: hot reset or link-down reset (LTSSM_STATUS: 0x0)
> [  852.483985] pci: PERST de-asserted by host!
> [  852.503041] pci: PERST asserted by host!
> [  852.521616] pci: PCIE_CLIENT_INTR_STATUS_MISC: 0x2003
> [  852.522375] pci: link up! (LTSSM_STATUS: 0x230011)
> [  852.610318] pci: PERST de-asserted by host!
> 
> The time between 845 and 852 is the time it takes for the RC to
> boot and probe the RC PCIe driver.
> 
> Note that I currently don't do anything in the perst gpio handler,
> I only print when receiving the PERST GPIO IRQ, as I wanted to be
> able to answer your question.
> 
> /
> Currently, what I do instead, in order to handle a link down
> (which clears all the RESBAR registers) followed by a link up,
> is to re-write the RESBAR registers when receiving the link down IRQ.
> (This is only to handle the case of the RC rebooting.)
> 
> A nicer solution would probably be to modify dw_pcie_ep_set_bar() to
> properly handle controllers with the Resizable BAR capability, and remove
> the RESBAR related code from dw_pcie_ep_init_complete().
> 
> However, that would still require changes in pci-epf-test.c to call
> set_bar() after a hot reset/link-down reset (and it is not possible
> to distinguish between them), which could be done by either:
> 1) Making sure that the glue drivers (that implement Resizable BAR capability)
>  call dw_pcie_ep_init_notify() when receiving a hot reset/link-down reset
>  IRQ (or maybe instead when getting the link up IRQ), as that will
>  trigger pci-epf-test.c to (once again) call set_bar().
> or
> 2) Modifying pci-epf-test.c:pci_epf_test_link_up() to call set_bar()
>  (If epc_features doesn't have a core_init_notifier, as in that case
>  set_bar() is called by pci_epf_test_core_init()).
>  (Since I assume that not all SoCs might have a PERST GPIO.)
> or
> 3) We could allow glue drivers that use an internal refclk to still make
>  use of the PERST GPIO IRQ, and only call dw_pcie_ep_init_notify(),
>  as that would also cause pci-epf-test.c to call set_bar().
>  (These glue drivers, which implement the Resizable BAR capability would
>  however not need to perform a full core reset, etc. in the PERST GPIO
>  IRQ handler, they only need to call dw_pcie_ep_init_notify().)
> 
> Right now, I think I prefer 1).
> 
> What do you think?
> 

[For this context, I'm not only focusing on REBAR but all of the non-sticky DWC
registers]

If the PCIe spec has mandated using PERST# for all endpoints, I would've
definitely went with option 3. But the spec has made PERST# as optional for the
endpoints, so we cannot force the glue drivers to make use of PERST# IRQ.

But the solution I'm thinking is, we should reconfigure all non-sticky DWC
registers during LINK_DOWN phase irrespective of whether core_init_notifier is
used or not. This should work for both cases because we can skip configuring the
DWC registers when the core_init platforms try to do it again during PERST#
deassert.

Wdyt?

- Mani

> Since it seems that not many SoCs that use a DWC core, have Resizable
> BAR capability implemented, I will try to see what I can come up with.
> 
> 
> Kind regards,
> Niklas

-- 
மணிவண்ணன் சதாசிவம்