[RFC,4/6] dma: Add Synopsys eDMA IP PCIe glue-logic

[Andy Shevchenko <andriy.shevchenko@linux.intel.com>]

On Thu, Dec 13, 2018 at 02:40:40PM +0000, Gustavo Pimentel wrote:
> On 12/12/2018 13:25, Andy Shevchenko wrote:
> > On Wed, Dec 12, 2018 at 12:13:24PM +0100, Gustavo Pimentel wrote:

> > It seems this driver somehow written as a copy-paste of existing pieces w/o good reasons to do such.
> 

(1)

> What do you mean? It's true I relied on existing drivers to develop this
> solution, but is not it supposed to be so?

See below.

> >> +enum dw_edma_pcie_bar {
> >> +	BAR_0,
> >> +	BAR_1,
> >> +	BAR_2,
> >> +	BAR_3,
> >> +	BAR_4,
> >> +	BAR_5
> >> +};
> > 
> > Why do you need this at all?
> 
> See my answer below.
> 
> > 
> >> +static const struct dw_edma_pcie_data snps_edda_data = {
> >> +	// eDMA registers location
> >> +	.regs_bar			= BAR_0,
> >> +	.regs_off			= 0x1000,	//   4 KBytes
> >> +	// eDMA memory linked list location
> >> +	.ll_bar				= BAR_2,
> >> +	.ll_off				= 0,		//   0 KBytes
> >> +	.ll_sz				= 0x20000,	// 128 KBytes
> >> +	// Other
> >> +	.version			= 0,
> >> +	.mode				= EDMA_MODE_UNROLL,
> >> +};
> > 
> > Huh? Isn't this 
> 
> The eDMA is a hardware block (with requires some configuration) accessible
> through a PCIe EndPoint, unfortunately there isn't any way for now (at least)
> through an PCIe capability for instance to detect those configurations
> automatically, therefore the only way to pass that configuration is to associate
> it to PCIe Vendor(0x16c3) and Device (0xedda) Id.
> 
> To interact with eDMA hardware block the driver needs to access the eDMA
> registers and the only way to do it is through PCIe mapping. In this those
> registers will be available on BAR 0, with a offset of 4KB from the start.
> 
> The driver also requires a memory space (RAM) to create a linked list
> descriptors and pass the first descriptor address to the eDMA hardware block so
> that it can start performing the transfer autonomously. Once again this memory
> space is provide through PCIe on BAR 2. This linked list space is in the
> beginning and it's restricted to a size of 128KB.
> 
> Since this eDMA hardware block (more specifically eDMA registers) can evolve in
> the future, I choosed to put here a version in order to have a driver that can
> be easily adaptable and reusable without much effort.

(2)

> >> +	err = pci_try_set_mwi(pdev);
> >> +	if (err) {
> >> +		dev_err(dev, "%s DMA memory write invalidate\n",
> >> +			pci_name(pdev));
> >> +		return err;
> >> +	}
> > 
> > Are you sure you need this?
> 
> I'm not sure, probably not. I saw using this being use on /dma/dw/pci.c driver.
> I'll test without it.

That's exactly the point what I referred to in (1) above.
Looks like here is cargo-cult programming done without understanding why.
I dunno how many other cases in the driver like this.

> >> +	dev_info(dev, "Mode:\t%s\n",
> >> +		 dw->mode == EDMA_MODE_LEGACY ? "Legacy" : "Unroll");
> >> +
> > 
> > 
> >> +	dev_info(dev, "Registers:\tBAR=%u, off=0x%.16llx B, addr=0x%.8lx\n",
> >> +		 pdata->regs_bar, pdata->regs_off,
> >> +		 (unsigned long) dw->regs);
> > 
> > Oh, no, don't do casting when printing something. In only rare cases it's needed, not here.
> 
> I've put the cast to remove the following warning:
> 
> drivers/dma/dw-edma/dw-edma-pcie.c:138:15:  warning: format ‘%lx’ expects
> argument of type ‘long unsigned int’, but argument 6 has type ‘void *’

...and as I said this is wrong. Please, find suitable specifiers for your case.

> >> +	if (pdev->msi_cap && pdev->msi_enabled) {
> >> +		cap_off = pdev->msi_cap + PCI_MSI_FLAGS;
> >> +		pci_read_config_word(pdev, cap_off, &flags);
> >> +		if (flags & PCI_MSI_FLAGS_ENABLE) {
> >> +			cap_off = pdev->msi_cap + PCI_MSI_ADDRESS_LO;
> >> +			pci_read_config_dword(pdev, cap_off, &addr_lo);
> >> +
> >> +			if (flags & PCI_MSI_FLAGS_64BIT) {
> >> +				cap_off = pdev->msi_cap + PCI_MSI_ADDRESS_HI;
> >> +				pci_read_config_dword(pdev, cap_off, &addr_hi);
> >> +				cap_off = pdev->msi_cap + PCI_MSI_DATA_64;
> >> +			} else {
> >> +				addr_hi = 0;
> >> +				cap_off = pdev->msi_cap + PCI_MSI_DATA_32;
> >> +			}
> >> +
> >> +			dw->msi_addr = addr_hi;
> >> +			dw->msi_addr <<= 32;
> >> +			dw->msi_addr |= addr_lo;
> >> +
> >> +			pci_read_config_dword(pdev, cap_off, &(dw->msi_data));
> >> +			dw->msi_data &= 0xffff;
> >> +
> >> +			dev_info(dev,
> >> +				 "MSI:\t\taddr=0x%.16llx, data=0x%.8x, nr=%d\n",
> >> +				 dw->msi_addr, dw->msi_data, pdev->irq);
> >> +		}
> >> +	}
> >> +
> >> +	if (pdev->msix_cap && pdev->msix_enabled) {
> >> +		u32 offset;
> >> +		u8 bir;
> >> +
> >> +		cap_off = pdev->msix_cap + PCI_MSIX_FLAGS;
> >> +		pci_read_config_word(pdev, cap_off, &flags);
> >> +
> >> +		if (flags & PCI_MSIX_FLAGS_ENABLE) {
> >> +			cap_off = pdev->msix_cap + PCI_MSIX_TABLE;
> >> +			pci_read_config_dword(pdev, cap_off, &offset);
> >> +
> >> +			bir = offset & PCI_MSIX_TABLE_BIR;
> >> +			offset &= PCI_MSIX_TABLE_OFFSET;
> >> +
> >> +			reg = pcim_iomap_table(pdev)[bir];
> >> +			reg += offset;
> >> +
> >> +			addr_lo = readl(reg + PCI_MSIX_ENTRY_LOWER_ADDR);
> >> +			addr_hi = readl(reg + PCI_MSIX_ENTRY_UPPER_ADDR);
> >> +			dw->msi_addr = addr_hi;
> >> +			dw->msi_addr <<= 32;
> >> +			dw->msi_addr |= addr_lo;
> >> +
> >> +			dw->msi_data = readl(reg + PCI_MSIX_ENTRY_DATA);
> >> +
> >> +			dev_info(dev,
> >> +				 "MSI-X:\taddr=0x%.16llx, data=0x%.8x, nr=%d\n",
> >> +				 dw->msi_addr, dw->msi_data, pdev->irq);
> >> +		}
> >> +	}
> > 
> > What is this? Why?
> 
> I need to find the PCIe interrupt vector address and value (either for MSI or
> MSI-X, depends what the system has selected) in order to configure eDMA later to
> generate the (done or abort) interruptions.

It seems rather hackish way to do something which should be done by PCI/driver core.
But perhaps Bjorn and others familiar with PCI endpoint facility can share more.
Also this applies to (2) above.

> >> +	if (!pdev->msi_enabled && !pdev->msix_enabled) {
> 
> I'm not finding it. Can you tell me what it is?
> 
> Note: pci_msi_enabled() on msi.c returns if MSI has not been disabled by the
> command-line option pci=nomsi

Can you look a bit more carefully? For example, assume that 'dev' (sub)prefix
in the name of function would tell something about scope of application.

pci_dev_...() ?

> > There is a helper from PCI core for this.