Re: FSL DMA engine transfer to PCI memory

[Felix Radensky <felix@embedded-sol.com>]

Hi Ira, Scott

On 01/25/2011 12:26 AM, Ira W. Snyder wrote:
> On Mon, Jan 24, 2011 at 11:47:22PM +0200, Felix Radensky wrote:
>> Hi,
>>
>> I'm trying to use FSL DMA engine to perform DMA transfer from
>> memory buffer obtained by kmalloc() to PCI memory. This is on
>> custom board based on P2020 running linux-2.6.35. The PCI
>> device is Altera FPGA, connected directly to SoC PCI-E controller.
>>
>> 01:00.0 Unassigned class [ff00]: Altera Corporation Unknown device
>> 0004 (rev 01)
>>           Subsystem: Altera Corporation Unknown device 0004
>>           Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
>> ParErr- Stepping- SERR- FastB2B-
>>           Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
>>   >TAbort-<TAbort-<MAbort->SERR-<PERR-
>>           Interrupt: pin A routed to IRQ 16
>>           Region 0: Memory at c0000000 (32-bit, non-prefetchable)
>> [size=128K]
>>           Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
>> Queue=0/0 Enable-
>>                   Address: 0000000000000000  Data: 0000
>>           Capabilities: [78] Power Management version 3
>>                   Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>                   Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>           Capabilities: [80] Express Endpoint IRQ 0
>>                   Device: Supported: MaxPayload 256 bytes, PhantFunc 0,
>> ExtTag-
>>                   Device: Latency L0s<64ns, L1<1us
>>                   Device: AtnBtn- AtnInd- PwrInd-
>>                   Device: Errors: Correctable- Non-Fatal- Fatal-
>> Unsupported-
>>                   Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>                   Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
>>                   Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 1
>>                   Link: Latency L0s unlimited, L1 unlimited
>>                   Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
>>                   Link: Speed 2.5Gb/s, Width x1
>>           Capabilities: [100] Virtual Channel
>>
>>
>> I can successfully writel() to PCI memory via address obtained from
>> pci_ioremap_bar().
>> Here's my DMA transfer routine
>>
>> static int dma_transfer(struct dma_chan *chan, void *dst, void *src,
>> size_t len)
>> {
>>       int rc = 0;
>>       dma_addr_t dma_src;
>>       dma_addr_t dma_dst;
>>       dma_cookie_t cookie;
>>       struct completion cmp;
>>       enum dma_status status;
>>       enum dma_ctrl_flags flags = 0;
>>       struct dma_device *dev = chan->device;
>>       struct dma_async_tx_descriptor *tx = NULL;
>>       unsigned long tmo = msecs_to_jiffies(FPGA_DMA_TIMEOUT_MS);
>>
>>       dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE);
>>       if (dma_mapping_error(dev->dev, dma_src)) {
>>           printk(KERN_ERR "Failed to map src for DMA\n");
>>           return -EIO;
>>       }
>>
>>       dma_dst = (dma_addr_t)dst;
>>
>>       flags = DMA_CTRL_ACK |
>>           DMA_COMPL_SRC_UNMAP_SINGLE  |
>>           DMA_COMPL_SKIP_DEST_UNMAP |
>>           DMA_PREP_INTERRUPT;
>>
>>       tx = dev->device_prep_dma_memcpy(chan, dma_dst, dma_src, len, flags);
>>       if (!tx) {
>>           printk(KERN_ERR "%s: Failed to prepare DMA transfer\n",
>>                  __FUNCTION__);
>>           dma_unmap_single(dev->dev, dma_src, len, DMA_TO_DEVICE);
>>           return -ENOMEM;
>>       }
>>
>>       init_completion(&cmp);
>>       tx->callback = dma_callback;
>>       tx->callback_param =&cmp;
>>       cookie = tx->tx_submit(tx);
>>
>>       if (dma_submit_error(cookie)) {
>>           printk(KERN_ERR "%s: Failed to start DMA transfer\n",
>>                  __FUNCTION__);
>>           return -ENOMEM;
>>       }
>>
>>       dma_async_issue_pending(chan);
>>
>>       tmo = wait_for_completion_timeout(&cmp, tmo);
>>       status = dma_async_is_tx_complete(chan, cookie, NULL, NULL);
>>
>>       if (tmo == 0) {
>>           printk(KERN_ERR "%s: Transfer timed out\n", __FUNCTION__);
>>           rc = -ETIMEDOUT;
>>       } else if (status != DMA_SUCCESS) {
>>           printk(KERN_ERR "%s: Transfer failed: status is %s\n",
>>                  __FUNCTION__,
>>                  status == DMA_ERROR ? "error" : "in progress");
>>
>>           dev->device_control(chan, DMA_TERMINATE_ALL, 0);
>>           rc = -EIO;
>>       }
>>
>>       return rc;
>> }
>>
>> The destination address is PCI memory address returned by
>> pci_ioremap_bar().
>> The transfer silently fails, destination buffer doesn't change
>> contents, but no
>> error condition is reported.
>>
>> What am I doing wrong ?
>>
>> Thanks a lot in advance.
>>
> Your destination address is wrong. The device_prep_dma_memcpy() routine
> works in physical addresses only (dma_addr_t type). Your source address
> looks fine: you're using the result of dma_map_single(), which returns a
> physical address.
>
> Your destination address should be something that comes from struct
> pci_dev.resource[x].start + offset if necessary. In your lspci output
> above, that will be 0xc0000000.
>
> Another possible problem: AFAIK you must use the _ONSTACK() variants
> from include/linux/completion.h for struct completion which are on the
> stack.
>
> Hope it helps,
> Ira

Thanks for your help. I'm now passing the result of 
pci_resource_start(pdev, 0)
as destination address, and destination buffer changes after the 
transfer. But
the contents of source and destination buffers are different. What 
else could
be wrong ?

Thanks.

Felix.