Re: FSL DMA engine transfer to PCI memory

[Felix Radensky <felix@embedded-sol.com>]

Hi Ira,

On 01/25/2011 02:18 AM, Ira W. Snyder wrote:
> On Tue, Jan 25, 2011 at 01:39:39AM +0200, Felix Radensky wrote:
>> Hi Ira, Scott
>>
>> On 01/25/2011 12:26 AM, Ira W. Snyder wrote:
>>> On Mon, Jan 24, 2011 at 11:47:22PM +0200, Felix Radensky wrote:
>>>> Hi,
>>>>
>>>> I'm trying to use FSL DMA engine to perform DMA transfer from
>>>> memory buffer obtained by kmalloc() to PCI memory. This is on
>>>> custom board based on P2020 running linux-2.6.35. The PCI
>>>> device is Altera FPGA, connected directly to SoC PCI-E controller.
>>>>
>>>> 01:00.0 Unassigned class [ff00]: Altera Corporation Unknown device
>>>> 0004 (rev 01)
>>>>            Subsystem: Altera Corporation Unknown device 0004
>>>>            Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop-
>>>> ParErr- Stepping- SERR- FastB2B-
>>>>            Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast
>>>>    >TAbort-<TAbort-<MAbort->SERR-<PERR-
>>>>            Interrupt: pin A routed to IRQ 16
>>>>            Region 0: Memory at c0000000 (32-bit, non-prefetchable)
>>>> [size=128K]
>>>>            Capabilities: [50] Message Signalled Interrupts: Mask- 64bit+
>>>> Queue=0/0 Enable-
>>>>                    Address: 0000000000000000  Data: 0000
>>>>            Capabilities: [78] Power Management version 3
>>>>                    Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA
>>>> PME(D0-,D1-,D2-,D3hot-,D3cold-)
>>>>                    Status: D0 PME-Enable- DSel=0 DScale=0 PME-
>>>>            Capabilities: [80] Express Endpoint IRQ 0
>>>>                    Device: Supported: MaxPayload 256 bytes, PhantFunc 0,
>>>> ExtTag-
>>>>                    Device: Latency L0s<64ns, L1<1us
>>>>                    Device: AtnBtn- AtnInd- PwrInd-
>>>>                    Device: Errors: Correctable- Non-Fatal- Fatal-
>>>> Unsupported-
>>>>                    Device: RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+
>>>>                    Device: MaxPayload 128 bytes, MaxReadReq 512 bytes
>>>>                    Link: Supported Speed 2.5Gb/s, Width x1, ASPM L0s, Port 1
>>>>                    Link: Latency L0s unlimited, L1 unlimited
>>>>                    Link: ASPM Disabled RCB 64 bytes CommClk- ExtSynch-
>>>>                    Link: Speed 2.5Gb/s, Width x1
>>>>            Capabilities: [100] Virtual Channel
>>>>
>>>>
>>>> I can successfully writel() to PCI memory via address obtained from
>>>> pci_ioremap_bar().
>>>> Here's my DMA transfer routine
>>>>
>>>> static int dma_transfer(struct dma_chan *chan, void *dst, void *src,
>>>> size_t len)
>>>> {
>>>>        int rc = 0;
>>>>        dma_addr_t dma_src;
>>>>        dma_addr_t dma_dst;
>>>>        dma_cookie_t cookie;
>>>>        struct completion cmp;
>>>>        enum dma_status status;
>>>>        enum dma_ctrl_flags flags = 0;
>>>>        struct dma_device *dev = chan->device;
>>>>        struct dma_async_tx_descriptor *tx = NULL;
>>>>        unsigned long tmo = msecs_to_jiffies(FPGA_DMA_TIMEOUT_MS);
>>>>
>>>>        dma_src = dma_map_single(dev->dev, src, len, DMA_TO_DEVICE);
>>>>        if (dma_mapping_error(dev->dev, dma_src)) {
>>>>            printk(KERN_ERR "Failed to map src for DMA\n");
>>>>            return -EIO;
>>>>        }
>>>>
>>>>        dma_dst = (dma_addr_t)dst;
>>>>
>>>>        flags = DMA_CTRL_ACK |
>>>>            DMA_COMPL_SRC_UNMAP_SINGLE  |
>>>>            DMA_COMPL_SKIP_DEST_UNMAP |
>>>>            DMA_PREP_INTERRUPT;
>>>>
>>>>        tx = dev->device_prep_dma_memcpy(chan, dma_dst, dma_src, len, flags);
>>>>        if (!tx) {
>>>>            printk(KERN_ERR "%s: Failed to prepare DMA transfer\n",
>>>>                   __FUNCTION__);
>>>>            dma_unmap_single(dev->dev, dma_src, len, DMA_TO_DEVICE);
>>>>            return -ENOMEM;
>>>>        }
>>>>
>>>>        init_completion(&cmp);
>>>>        tx->callback = dma_callback;
>>>>        tx->callback_param =&cmp;
>>>>        cookie = tx->tx_submit(tx);
>>>>
>>>>        if (dma_submit_error(cookie)) {
>>>>            printk(KERN_ERR "%s: Failed to start DMA transfer\n",
>>>>                   __FUNCTION__);
>>>>            return -ENOMEM;
>>>>        }
>>>>
>>>>        dma_async_issue_pending(chan);
>>>>
>>>>        tmo = wait_for_completion_timeout(&cmp, tmo);
>>>>        status = dma_async_is_tx_complete(chan, cookie, NULL, NULL);
>>>>
>>>>        if (tmo == 0) {
>>>>            printk(KERN_ERR "%s: Transfer timed out\n", __FUNCTION__);
>>>>            rc = -ETIMEDOUT;
>>>>        } else if (status != DMA_SUCCESS) {
>>>>            printk(KERN_ERR "%s: Transfer failed: status is %s\n",
>>>>                   __FUNCTION__,
>>>>                   status == DMA_ERROR ? "error" : "in progress");
>>>>
>>>>            dev->device_control(chan, DMA_TERMINATE_ALL, 0);
>>>>            rc = -EIO;
>>>>        }
>>>>
>>>>        return rc;
>>>> }
>>>>
>>>> The destination address is PCI memory address returned by
>>>> pci_ioremap_bar().
>>>> The transfer silently fails, destination buffer doesn't change
>>>> contents, but no
>>>> error condition is reported.
>>>>
>>>> What am I doing wrong ?
>>>>
>>>> Thanks a lot in advance.
>>>>
>>> Your destination address is wrong. The device_prep_dma_memcpy() routine
>>> works in physical addresses only (dma_addr_t type). Your source address
>>> looks fine: you're using the result of dma_map_single(), which returns a
>>> physical address.
>>>
>>> Your destination address should be something that comes from struct
>>> pci_dev.resource[x].start + offset if necessary. In your lspci output
>>> above, that will be 0xc0000000.
>>>
>>> Another possible problem: AFAIK you must use the _ONSTACK() variants
>>> from include/linux/completion.h for struct completion which are on the
>>> stack.
>>>
>>> Hope it helps,
>>> Ira
>> Thanks for your help. I'm now passing the result of
>> pci_resource_start(pdev, 0)
>> as destination address, and destination buffer changes after the
>> transfer. But
>> the contents of source and destination buffers are different. What
>> else could
>> be wrong ?
>>
> After you changed the dst address to pci_resource_start(pdev, 0), I
> don't see anything wrong with the code.
>
> Try using memcpy_toio() to copy some bytes to the FPGA. Also try writing
> a single byte at a time (writeb()?) in a loop. This should help
> establish that your device is working.
>
> If you put some pattern in your src buffer (such as 0x0, 0x1, 0x2, ...
> 0xff, repeat) does the destination show some pattern after the DMA
> completes? (Such as, every 4th byte is correct.)
>
> Ira

memcpy_toio() works fine, the data is written correctly. After
DMA, the correct data appears at offsets 0xC, 0x1C, 0x2C, etc.
of the destination buffer. I have 12 bytes of junk, 4 bytes of
correct data, then again 12 bytes of junk and so on.

Felix.