Re: [PATCH v3 9/9] PCI: endpoint: Set prefetch when allocating memory for 64-bit BARs

[Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>]

+ Arnd

On Wed, Mar 13, 2024 at 11:58:01AM +0100, Niklas Cassel wrote:
> From the PCIe 6.0 base spec:

It'd be good to mention the section also.

> "Generally only 64-bit BARs are good candidates, since only Legacy
> Endpoints are permitted to set the Prefetchable bit in 32-bit BARs,
> and most scalable platforms map all 32-bit Memory BARs into
> non-prefetchable Memory Space regardless of the Prefetchable bit value."
> 
> "For a PCI Express Endpoint, 64-bit addressing must be supported for all
> BARs that have the Prefetchable bit Set. 32-bit addressing is permitted
> for all BARs that do not have the Prefetchable bit Set."
> 
> "Any device that has a range that behaves like normal memory should mark
> the range as prefetchable. A linear frame buffer in a graphics device is
> an example of a range that should be marked prefetchable."
> 
> The PCIe spec tells us that we should have the prefetchable bit set for
> 64-bit BARs backed by "normal memory". The backing memory that we allocate
> for a 64-bit BAR using pci_epf_alloc_space() (which calls
> dma_alloc_coherent()) is obviously "normal memory".
> 

I'm not sure this is correct. Memory returned by 'dma_alloc_coherent' is not the
'normal memory' but rather 'consistent/coherent memory'. Here the question is,
can the memory returned by dma_alloc_coherent() be prefetched or write-combined
on all architectures.

I hope Arnd can answer this question.

- Mani

> Thus, set the prefetchable bit when allocating backing memory for a 64-bit
> BAR.
> 
> Signed-off-by: Niklas Cassel <cassel@kernel.org>
> ---
>  drivers/pci/endpoint/pci-epf-core.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/pci/endpoint/pci-epf-core.c b/drivers/pci/endpoint/pci-epf-core.c
> index e7dbbeb1f0de..20d2bde0747c 100644
> --- a/drivers/pci/endpoint/pci-epf-core.c
> +++ b/drivers/pci/endpoint/pci-epf-core.c
> @@ -309,6 +309,9 @@ void *pci_epf_alloc_space(struct pci_epf *epf, size_t size, enum pci_barno bar,
>  	else
>  		epf_bar[bar].flags |= PCI_BASE_ADDRESS_MEM_TYPE_32;
>  
> +	if (epf_bar[bar].flags & PCI_BASE_ADDRESS_MEM_TYPE_64)
> +		epf_bar[bar].flags |= PCI_BASE_ADDRESS_MEM_PREFETCH;
> +
>  	return space;
>  }
>  EXPORT_SYMBOL_GPL(pci_epf_alloc_space);
> -- 
> 2.44.0
> 

-- 
மணிவண்ணன் சதாசிவம்