Re: [PATCH V4] PCI: rcar: Add L1 link state fix into data abort hook

[Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>]

On Sun, Nov 29, 2020 at 02:05:08PM +0100, Marek Vasut wrote:
> On 11/19/20 6:35 PM, Lorenzo Pieralisi wrote:
> > > +#ifdef CONFIG_ARM
> > > +/*
> > > + * Here we keep a static copy of the remapped PCIe controller address.
> > > + * This is only used on aarch32 systems, all of which have one single
> > > + * PCIe controller, to provide quick access to the PCIe controller in
> > > + * the L1 link state fixup function, called from the ARM fault handler.
> > > + */
> > > +static void __iomem *pcie_base;
> > > +/*
> > > + * Static copy of bus clock pointer, so we can check whether the clock
> > > + * is enabled or not.
> > > + */
> > > +static struct clk *pcie_bus_clk;
> > > +#endif
> > 
> > Don't think you can have multiple host bridges in a given platform,
> > if it is a possible configuration this won't work.
> 
> Correct, all the affected platforms have only one host bridge.
> 
> > >   static inline struct rcar_msi *to_rcar_msi(struct msi_controller *chip)
> > >   {
> > >   	return container_of(chip, struct rcar_msi, chip);
> > > @@ -804,6 +820,12 @@ static int rcar_pcie_get_resources(struct rcar_pcie_host *host)
> > >   	}
> > >   	host->msi.irq2 = i;
> > > +#ifdef CONFIG_ARM
> > > +	/* Cache static copy for L1 link state fixup hook on aarch32 */
> > > +	pcie_base = pcie->base;
> > > +	pcie_bus_clk = host->bus_clk;
> > > +#endif
> > > +
> > >   	return 0;
> > >   err_irq2:
> > > @@ -1050,4 +1072,58 @@ static struct platform_driver rcar_pcie_driver = {
> > >   	},
> > >   	.probe = rcar_pcie_probe,
> > >   };
> > > +
> > > +#ifdef CONFIG_ARM
> > > +static int rcar_pcie_aarch32_abort_handler(unsigned long addr,
> > > +		unsigned int fsr, struct pt_regs *regs)
> > > +{
> > > +	u32 pmsr;
> > > +
> > > +	if (!pcie_base || !__clk_is_enabled(pcie_bus_clk))
> > > +		return 1;
> > > +
> > > +	pmsr = readl(pcie_base + PMSR);
> > > +
> > > +	/*
> > > +	 * Test if the PCIe controller received PM_ENTER_L1 DLLP and
> > > +	 * the PCIe controller is not in L1 link state. If true, apply
> > > +	 * fix, which will put the controller into L1 link state, from
> > > +	 * which it can return to L0s/L0 on its own.
> > > +	 */
> > > +	if ((pmsr & PMEL1RX) && ((pmsr & PMSTATE) != PMSTATE_L1)) {
> > > +		writel(L1IATN, pcie_base + PMCTLR);
> > > +		while (!(readl(pcie_base + PMSR) & L1FAEG))
> > > +			;
> > > +		writel(L1FAEG | PMEL1RX, pcie_base + PMSR);
> > > +		return 0;
> > > +	}
> > 
> > I suppose a fault on multiple cores can happen simultaneously, if it
> > does this may not work well either - I assume all config/io/mem would
> > trigger a fault.
> > 
> > As I mentioned in my reply to v1, is there a chance we can move
> > this quirk into config accessors (if the PM_ENTER_L1_DLLP is
> > subsequent to a write into PMCSR to programme a D state) ?
> 
> I don't think we can, since the userspace can do such a config space write
> with e.g. setpci and then this fixup is still needed.


Userspace goes via the kernel config accessors anyway, right ?

I would like to avoid having arch specific hooks in PCI drivers so
if we can work around it somehow it is much better.

I can still merge this patch this week but I would like to explore
alternatives before committing it.

Lorenzo
> 
> > Config access is serialized but I suspect as I said above that this
> > triggers on config/io/mem alike.
> > 
> > Just asking to try to avoid a fault handler if possible.
> 
> See above, I doubt we can fully avoid this workaround.
> 
> [...]