* [v2,2/7] dt-bindings: stm32-dmamux: Add one cell to support DMA/MDMA chain
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
Add one cell to support DMA/MDMA chaining.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
Documentation/devicetree/bindings/dma/stm32-dmamux.txt | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
index 1b893b2..5e92b59 100644
--- a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
+++ b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
@@ -4,9 +4,9 @@ Required properties:
- compatible: "st,stm32h7-dmamux"
- reg: Memory map for accessing module
- #dma-cells: Should be set to <3>.
- First parameter is request line number.
- Second is DMA channel configuration
- Third is Fifo threshold
+- First parameter is request line number.
+- Second is DMA channel configuration
+- Third is a 32bit bitfield
For more details about the three cells, please see
stm32-dma.txt documentation binding file
- dma-masters: Phandle pointing to the DMA controllers.
^ permalink raw reply related
* [v2,3/7] dt-bindings: stm32-mdma: Add DMA/MDMA chaining support bindings
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
This patch adds the description of the 2 properties needed to support M2M
transfer triggered by STM32 DMA when his transfer is complete.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
Documentation/devicetree/bindings/dma/stm32-mdma.txt | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/Documentation/devicetree/bindings/dma/stm32-mdma.txt b/Documentation/devicetree/bindings/dma/stm32-mdma.txt
index d18772d..27c2812 100644
--- a/Documentation/devicetree/bindings/dma/stm32-mdma.txt
+++ b/Documentation/devicetree/bindings/dma/stm32-mdma.txt
@@ -10,7 +10,7 @@ Required properties:
- interrupts: Should contain the MDMA interrupt.
- clocks: Should contain the input clock of the DMA instance.
- resets: Reference to a reset controller asserting the DMA controller.
-- #dma-cells : Must be <5>. See DMA client paragraph for more details.
+- #dma-cells : Must be <6>. See DMA client paragraph for more details.
Optional properties:
- dma-channels: Number of DMA channels supported by the controller.
@@ -26,7 +26,7 @@ Example:
interrupts = <122>;
clocks = <&timer_clk>;
resets = <&rcc 992>;
- #dma-cells = <5>;
+ #dma-cells = <6>;
dma-channels = <16>;
dma-requests = <32>;
st,ahb-addr-masks = <0x20000000>, <0x00000000>;
@@ -35,8 +35,8 @@ Example:
* DMA client
DMA clients connected to the STM32 MDMA controller must use the format
-described in the dma.txt file, using a five-cell specifier for each channel:
-a phandle to the MDMA controller plus the following five integer cells:
+described in the dma.txt file, using a six-cell specifier for each channel:
+a phandle to the MDMA controller plus the following six integer cells:
1. The request line number
2. The priority level
@@ -76,6 +76,10 @@ a phandle to the MDMA controller plus the following five integer cells:
if no HW ack signal is used by the MDMA client
5. A 32bit mask specifying the value to be written to acknowledge the request
if no HW ack signal is used by the MDMA client
+6. A bitfield value specifying if the MDMA client wants to generate M2M
+ transfer with HW trigger (1) or not (0). This bitfield should be only
+ enabled for M2M transfer triggered by STM32 DMA client. The memory devices
+ involved in this kind of transfer are SRAM and DDR.
Example:
^ permalink raw reply related
* [v2,4/7] dmaengine: stm32-dma: Add DMA/MDMA chaining support
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
This patch adds support of DMA/MDMA chaining support.
It introduces an intermediate transfer between peripherals and STM32 DMA.
This intermediate transfer is triggered by SW for single M2D transfer and
by STM32 DMA IP for all other modes (sg, cyclic) and direction (D2M).
A generic SRAM allocator is used for this intermediate buffer
Each DMA channel will be able to define its SRAM needs to achieve chaining
feature : (2 ^ order) * PAGE_SIZE.
For cyclic, SRAM buffer is derived from period length (rounded on
PAGE_SIZE).
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
drivers/dma/stm32-dma.c | 879 ++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 772 insertions(+), 107 deletions(-)
diff --git a/drivers/dma/stm32-dma.c b/drivers/dma/stm32-dma.c
index 379e8d5..1571f2f 100644
--- a/drivers/dma/stm32-dma.c
+++ b/drivers/dma/stm32-dma.c
@@ -15,11 +15,14 @@
#include <linux/dmaengine.h>
#include <linux/dma-mapping.h>
#include <linux/err.h>
+#include <linux/genalloc.h>
#include <linux/init.h>
+#include <linux/iopoll.h>
#include <linux/jiffies.h>
#include <linux/list.h>
#include <linux/module.h>
#include <linux/of.h>
+#include <linux/of_address.h>
#include <linux/of_device.h>
#include <linux/of_dma.h>
#include <linux/platform_device.h>
@@ -118,6 +121,7 @@
#define STM32_DMA_FIFO_THRESHOLD_FULL 0x03
#define STM32_DMA_MAX_DATA_ITEMS 0xffff
+#define STM32_DMA_SRAM_GRANULARITY PAGE_SIZE
/*
* Valid transfer starts from @0 to @0xFFFE leading to unaligned scatter
* gather at boundary. Thus it's safer to round down this value on FIFO
@@ -135,6 +139,12 @@
/* DMA Features */
#define STM32_DMA_THRESHOLD_FTR_MASK GENMASK(1, 0)
#define STM32_DMA_THRESHOLD_FTR_GET(n) ((n) & STM32_DMA_THRESHOLD_FTR_MASK)
+#define STM32_DMA_MDMA_CHAIN_FTR_MASK BIT(2)
+#define STM32_DMA_MDMA_CHAIN_FTR_GET(n) (((n) & STM32_DMA_MDMA_CHAIN_FTR_MASK) \
+ >> 2)
+#define STM32_DMA_MDMA_SRAM_SIZE_MASK GENMASK(4, 3)
+#define STM32_DMA_MDMA_SRAM_SIZE_GET(n) (((n) & STM32_DMA_MDMA_SRAM_SIZE_MASK) \
+ >> 3)
enum stm32_dma_width {
STM32_DMA_BYTE,
@@ -176,15 +186,31 @@ struct stm32_dma_chan_reg {
u32 dma_sfcr;
};
+struct stm32_dma_mdma_desc {
+ struct sg_table sgt;
+ struct dma_async_tx_descriptor *desc;
+};
+
+struct stm32_dma_mdma {
+ struct dma_chan *chan;
+ enum dma_transfer_direction dir;
+ dma_addr_t sram_buf;
+ u32 sram_period;
+ u32 num_sgs;
+};
+
struct stm32_dma_sg_req {
- u32 len;
+ struct scatterlist stm32_sgl_req;
struct stm32_dma_chan_reg chan_reg;
+ struct stm32_dma_mdma_desc m_desc;
};
struct stm32_dma_desc {
struct virt_dma_desc vdesc;
bool cyclic;
u32 num_sgs;
+ dma_addr_t dma_buf;
+ void *dma_buf_cpu;
struct stm32_dma_sg_req sg_req[];
};
@@ -201,6 +227,10 @@ struct stm32_dma_chan {
u32 threshold;
u32 mem_burst;
u32 mem_width;
+ struct stm32_dma_mdma mchan;
+ u32 use_mdma;
+ u32 sram_size;
+ u32 residue_after_drain;
};
struct stm32_dma_device {
@@ -210,6 +240,7 @@ struct stm32_dma_device {
struct reset_control *rst;
bool mem2mem;
struct stm32_dma_chan chan[STM32_DMA_MAX_CHANNELS];
+ struct gen_pool *sram_pool;
};
static struct stm32_dma_device *stm32_dma_get_dev(struct stm32_dma_chan *chan)
@@ -497,11 +528,15 @@ static void stm32_dma_stop(struct stm32_dma_chan *chan)
static int stm32_dma_terminate_all(struct dma_chan *c)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
unsigned long flags;
LIST_HEAD(head);
spin_lock_irqsave(&chan->vchan.lock, flags);
+ if (chan->use_mdma)
+ dmaengine_terminate_async(mchan->chan);
+
if (chan->busy) {
stm32_dma_stop(chan);
chan->desc = NULL;
@@ -514,9 +549,96 @@ static int stm32_dma_terminate_all(struct dma_chan *c)
return 0;
}
+static u32 stm32_dma_get_remaining_bytes(struct stm32_dma_chan *chan)
+{
+ u32 dma_scr, width, ndtr;
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+
+ dma_scr = stm32_dma_read(dmadev, STM32_DMA_SCR(chan->id));
+ width = STM32_DMA_SCR_PSIZE_GET(dma_scr);
+ ndtr = stm32_dma_read(dmadev, STM32_DMA_SNDTR(chan->id));
+
+ return ndtr << width;
+}
+
+static int stm32_dma_mdma_drain(struct stm32_dma_chan *chan)
+{
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+ struct stm32_dma_sg_req *sg_req;
+ struct dma_device *ddev = mchan->chan->device;
+ struct dma_async_tx_descriptor *desc = NULL;
+ enum dma_status status;
+ dma_addr_t src_buf, dst_buf;
+ u32 mdma_residue, mdma_wrote, dma_to_write, len;
+ struct dma_tx_state state;
+ int ret;
+
+ /* DMA/MDMA chain: drain remaining data in SRAM */
+
+ /* Get the residue on MDMA side */
+ status = dmaengine_tx_status(mchan->chan, mchan->chan->cookie, &state);
+ if (status == DMA_COMPLETE)
+ return status;
+
+ mdma_residue = state.residue;
+ sg_req = &chan->desc->sg_req[chan->next_sg - 1];
+ len = sg_dma_len(&sg_req->stm32_sgl_req);
+
+ /*
+ * Total = mdma blocks * sram_period + rest (< sram_period)
+ * so mdma blocks * sram_period = len - mdma residue - rest
+ */
+ mdma_wrote = len - mdma_residue - (len % mchan->sram_period);
+
+ /* Remaining data stuck in SRAM */
+ dma_to_write = mchan->sram_period - stm32_dma_get_remaining_bytes(chan);
+ if (dma_to_write > 0) {
+ /* Stop DMA current operation */
+ stm32_dma_disable_chan(chan);
+
+ /* Terminate current MDMA to initiate a new one */
+ dmaengine_terminate_all(mchan->chan);
+
+ /* Double buffer management */
+ src_buf = mchan->sram_buf +
+ ((mdma_wrote / mchan->sram_period) & 0x1) *
+ mchan->sram_period;
+ dst_buf = sg_dma_address(&sg_req->stm32_sgl_req) + mdma_wrote;
+
+ desc = ddev->device_prep_dma_memcpy(mchan->chan,
+ dst_buf, src_buf,
+ dma_to_write,
+ DMA_PREP_INTERRUPT);
+ if (!desc)
+ return -EINVAL;
+
+ ret = dma_submit_error(dmaengine_submit(desc));
+ if (ret < 0)
+ return ret;
+
+ status = dma_wait_for_async_tx(desc);
+ if (status != DMA_COMPLETE) {
+ dev_err(chan2dev(chan), "flush() dma_wait_for_async_tx error\n");
+ dmaengine_terminate_async(mchan->chan);
+ return -EBUSY;
+ }
+
+ /* We need to store residue for tx_status() */
+ chan->residue_after_drain = len - (mdma_wrote + dma_to_write);
+ }
+
+ return 0;
+}
+
static void stm32_dma_synchronize(struct dma_chan *c)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+
+ if (chan->desc && chan->use_mdma && mchan->dir == DMA_DEV_TO_MEM)
+ if (stm32_dma_mdma_drain(chan))
+ dev_err(chan2dev(chan), "%s: can't drain DMA\n",
+ __func__);
vchan_synchronize(&chan->vchan);
}
@@ -539,62 +661,232 @@ static void stm32_dma_dump_reg(struct stm32_dma_chan *chan)
dev_dbg(chan2dev(chan), "SFCR: 0x%08x\n", sfcr);
}
-static void stm32_dma_configure_next_sg(struct stm32_dma_chan *chan);
-
-static void stm32_dma_start_transfer(struct stm32_dma_chan *chan)
+static int stm32_dma_dummy_memcpy_xfer(struct stm32_dma_chan *chan)
{
struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
- struct virt_dma_desc *vdesc;
+ struct dma_device *ddev = &dmadev->ddev;
+ struct stm32_dma_chan_reg reg;
+ u8 src_buf, dst_buf;
+ dma_addr_t dma_src_buf, dma_dst_buf;
+ u32 ndtr, status;
+ int len, ret;
+
+ ret = 0;
+ src_buf = 0;
+ len = 1;
+
+ dma_src_buf = dma_map_single(ddev->dev, &src_buf, len, DMA_TO_DEVICE);
+ ret = dma_mapping_error(ddev->dev, dma_src_buf);
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "Source buffer map failed\n");
+ return ret;
+ }
+
+ dma_dst_buf = dma_map_single(ddev->dev, &dst_buf, len, DMA_FROM_DEVICE);
+ ret = dma_mapping_error(ddev->dev, dma_dst_buf);
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "Destination buffer map failed\n");
+ dma_unmap_single(ddev->dev, dma_src_buf, len, DMA_TO_DEVICE);
+ return ret;
+ }
+
+ reg.dma_scr = STM32_DMA_SCR_DIR(STM32_DMA_MEM_TO_MEM) |
+ STM32_DMA_SCR_PBURST(STM32_DMA_BURST_SINGLE) |
+ STM32_DMA_SCR_MBURST(STM32_DMA_BURST_SINGLE) |
+ STM32_DMA_SCR_MINC |
+ STM32_DMA_SCR_PINC |
+ STM32_DMA_SCR_TEIE;
+ reg.dma_spar = dma_src_buf;
+ reg.dma_sm0ar = dma_dst_buf;
+ reg.dma_sfcr = STM32_DMA_SFCR_MASK |
+ STM32_DMA_SFCR_FTH(STM32_DMA_FIFO_THRESHOLD_FULL);
+ reg.dma_sm1ar = dma_dst_buf;
+ reg.dma_sndtr = 1;
+
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg.dma_scr);
+ stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg.dma_spar);
+ stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg.dma_sm0ar);
+ stm32_dma_write(dmadev, STM32_DMA_SFCR(chan->id), reg.dma_sfcr);
+ stm32_dma_write(dmadev, STM32_DMA_SM1AR(chan->id), reg.dma_sm1ar);
+ stm32_dma_write(dmadev, STM32_DMA_SNDTR(chan->id), reg.dma_sndtr);
+
+ /* Clear interrupt status if it is there */
+ status = stm32_dma_irq_status(chan);
+ if (status)
+ stm32_dma_irq_clear(chan, status);
+
+ stm32_dma_dump_reg(chan);
+
+ chan->busy = true;
+ /* Start DMA */
+ reg.dma_scr |= STM32_DMA_SCR_EN;
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg.dma_scr);
+
+ ret = readl_relaxed_poll_timeout_atomic(dmadev->base +
+ STM32_DMA_SNDTR(chan->id),
+ ndtr, !ndtr, 10, 1000);
+ if (ret) {
+ dev_err(chan2dev(chan), "%s: timeout!\n", __func__);
+ ret = -EBUSY;
+ }
+
+ chan->busy = false;
+
+ ret = stm32_dma_disable_chan(chan);
+ status = stm32_dma_irq_status(chan);
+ if (status)
+ stm32_dma_irq_clear(chan, status);
+
+ dma_unmap_single(ddev->dev, dma_src_buf, len, DMA_TO_DEVICE);
+ dma_unmap_single(ddev->dev, dma_dst_buf, len, DMA_FROM_DEVICE);
+
+ return ret;
+}
+
+static int stm32_dma_mdma_flush_remaining(struct stm32_dma_chan *chan)
+{
+ struct stm32_dma_mdma *mchan = &chan->mchan;
struct stm32_dma_sg_req *sg_req;
- struct stm32_dma_chan_reg *reg;
- u32 status;
+ struct dma_device *ddev = mchan->chan->device;
+ struct dma_async_tx_descriptor *desc = NULL;
+ enum dma_status status;
+ dma_addr_t src_buf, dst_buf;
+ u32 residue, remain, len;
int ret;
- ret = stm32_dma_disable_chan(chan);
- if (ret < 0)
- return;
+ sg_req = &chan->desc->sg_req[chan->next_sg - 1];
- if (!chan->desc) {
- vdesc = vchan_next_desc(&chan->vchan);
- if (!vdesc)
- return;
+ residue = stm32_dma_get_remaining_bytes(chan);
+ len = sg_dma_len(&sg_req->stm32_sgl_req);
+ remain = len % mchan->sram_period;
- chan->desc = to_stm32_dma_desc(vdesc);
- chan->next_sg = 0;
+ if (residue > 0 && len > mchan->sram_period &&
+ ((len % mchan->sram_period) != 0)) {
+ unsigned long dma_sync_wait_timeout =
+ jiffies + msecs_to_jiffies(5000);
+
+ while (residue > 0 &&
+ residue > (mchan->sram_period - remain)) {
+ if (time_after_eq(jiffies, dma_sync_wait_timeout)) {
+ dev_err(chan2dev(chan),
+ "%s timeout waiting for last bytes\n",
+ __func__);
+ break;
+ }
+ cpu_relax();
+ residue = stm32_dma_get_remaining_bytes(chan);
+ }
+ stm32_dma_disable_chan(chan);
+
+ src_buf = mchan->sram_buf + ((len / mchan->sram_period) & 0x1)
+ * mchan->sram_period;
+ dst_buf = sg_dma_address(&sg_req->stm32_sgl_req) + len -
+ (len % mchan->sram_period);
+
+ desc = ddev->device_prep_dma_memcpy(mchan->chan,
+ dst_buf, src_buf,
+ len % mchan->sram_period,
+ DMA_PREP_INTERRUPT);
+
+ if (!desc)
+ return -EINVAL;
+
+ ret = dma_submit_error(dmaengine_submit(desc));
+ if (ret < 0)
+ return ret;
+
+ status = dma_wait_for_async_tx(desc);
+ if (status != DMA_COMPLETE) {
+ dmaengine_terminate_async(mchan->chan);
+ return -EBUSY;
+ }
}
- if (chan->next_sg == chan->desc->num_sgs)
- chan->next_sg = 0;
+ return 0;
+}
- sg_req = &chan->desc->sg_req[chan->next_sg];
- reg = &sg_req->chan_reg;
+static void stm32_dma_start_transfer(struct stm32_dma_chan *chan);
- stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
- stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg->dma_spar);
- stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg->dma_sm0ar);
- stm32_dma_write(dmadev, STM32_DMA_SFCR(chan->id), reg->dma_sfcr);
- stm32_dma_write(dmadev, STM32_DMA_SM1AR(chan->id), reg->dma_sm1ar);
- stm32_dma_write(dmadev, STM32_DMA_SNDTR(chan->id), reg->dma_sndtr);
+static void stm32_mdma_chan_complete(void *param,
+ const struct dmaengine_result *result)
+{
+ struct stm32_dma_chan *chan = param;
- chan->next_sg++;
+ chan->busy = false;
+ if (result->result == DMA_TRANS_NOERROR) {
+ if (stm32_dma_mdma_flush_remaining(chan)) {
+ dev_err(chan2dev(chan), "Can't flush DMA\n");
+ return;
+ }
- /* Clear interrupt status if it is there */
- status = stm32_dma_irq_status(chan);
- if (status)
- stm32_dma_irq_clear(chan, status);
+ if (chan->next_sg == chan->desc->num_sgs) {
+ list_del(&chan->desc->vdesc.node);
+ vchan_cookie_complete(&chan->desc->vdesc);
+ chan->desc = NULL;
+ }
+ stm32_dma_start_transfer(chan);
+ } else {
+ dev_err(chan2dev(chan), "MDMA transfer error: %d\n",
+ result->result);
+ }
+}
- if (chan->desc->cyclic)
- stm32_dma_configure_next_sg(chan);
+static int stm32_dma_mdma_start(struct stm32_dma_chan *chan,
+ struct stm32_dma_sg_req *sg_req)
+{
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+ struct stm32_dma_mdma_desc *m_desc = &sg_req->m_desc;
+ struct dma_slave_config config;
+ int ret;
- stm32_dma_dump_reg(chan);
+ /* Configure MDMA channel */
+ memset(&config, 0, sizeof(config));
+ if (mchan->dir == DMA_MEM_TO_DEV)
+ config.dst_addr = mchan->sram_buf;
+ else
+ config.src_addr = mchan->sram_buf;
- /* Start DMA */
- reg->dma_scr |= STM32_DMA_SCR_EN;
- stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
+ ret = dmaengine_slave_config(mchan->chan, &config);
+ if (ret < 0)
+ goto error;
+
+ /* Prepare MDMA descriptor */
+ m_desc->desc = dmaengine_prep_slave_sg(mchan->chan, m_desc->sgt.sgl,
+ m_desc->sgt.nents, mchan->dir,
+ DMA_PREP_INTERRUPT);
+ if (!m_desc->desc) {
+ ret = -EINVAL;
+ goto error;
+ }
- chan->busy = true;
+ if (mchan->dir != DMA_MEM_TO_DEV) {
+ m_desc->desc->callback_result = stm32_mdma_chan_complete;
+ m_desc->desc->callback_param = chan;
+ }
- dev_dbg(chan2dev(chan), "vchan %pK: started\n", &chan->vchan);
+ ret = dma_submit_error(dmaengine_submit(m_desc->desc));
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "MDMA submit failed\n");
+ goto error;
+ }
+
+ dma_async_issue_pending(mchan->chan);
+
+ /*
+ * In case of M2D transfer, we have to generate dummy DMA transfer to
+ * copy 1st sg data into SRAM
+ */
+ if (mchan->dir == DMA_MEM_TO_DEV) {
+ ret = stm32_dma_dummy_memcpy_xfer(chan);
+ if (ret < 0) {
+ dmaengine_terminate_async(mchan->chan);
+ goto error;
+ }
+ }
+
+ return 0;
+error:
+ return ret;
}
static void stm32_dma_configure_next_sg(struct stm32_dma_chan *chan)
@@ -626,23 +918,132 @@ static void stm32_dma_configure_next_sg(struct stm32_dma_chan *chan)
}
}
-static void stm32_dma_handle_chan_done(struct stm32_dma_chan *chan)
+static void stm32_dma_start_transfer(struct stm32_dma_chan *chan)
{
- if (chan->desc) {
- if (chan->desc->cyclic) {
- vchan_cyclic_callback(&chan->desc->vdesc);
- chan->next_sg++;
- stm32_dma_configure_next_sg(chan);
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct virt_dma_desc *vdesc;
+ struct stm32_dma_sg_req *sg_req;
+ struct stm32_dma_chan_reg *reg;
+ u32 status;
+ int ret;
+
+ ret = stm32_dma_disable_chan(chan);
+ if (ret < 0)
+ return;
+
+ if (!chan->desc) {
+ vdesc = vchan_next_desc(&chan->vchan);
+ if (!vdesc)
+ return;
+
+ chan->desc = to_stm32_dma_desc(vdesc);
+ chan->next_sg = 0;
+ } else {
+ vdesc = &chan->desc->vdesc;
+ }
+
+ if (chan->next_sg == chan->desc->num_sgs)
+ chan->next_sg = 0;
+
+ sg_req = &chan->desc->sg_req[chan->next_sg];
+ reg = &sg_req->chan_reg;
+
+ /* Clear interrupt status if it is there */
+ status = stm32_dma_irq_status(chan);
+ if (status)
+ stm32_dma_irq_clear(chan, status);
+
+ if (chan->use_mdma) {
+ if (chan->next_sg == 0) {
+ struct stm32_dma_mdma_desc *m_desc;
+
+ m_desc = &sg_req->m_desc;
+ if (chan->desc->cyclic) {
+ /*
+ * If one callback is set, it will be called by
+ * MDMA driver.
+ */
+ if (vdesc->tx.callback) {
+ m_desc->desc->callback =
+ vdesc->tx.callback;
+ m_desc->desc->callback_param =
+ vdesc->tx.callback_param;
+ vdesc->tx.callback = NULL;
+ vdesc->tx.callback_param = NULL;
+ }
+ }
+ }
+
+ if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ ret = stm32_dma_dummy_memcpy_xfer(chan);
+ if (ret < 0) {
+ dmaengine_terminate_async(chan->mchan.chan);
+ chan->desc = NULL;
+ return;
+ }
} else {
- chan->busy = false;
- if (chan->next_sg == chan->desc->num_sgs) {
- list_del(&chan->desc->vdesc.node);
- vchan_cookie_complete(&chan->desc->vdesc);
+ reg->dma_scr &= ~STM32_DMA_SCR_TCIE;
+ }
+
+ if (!chan->desc->cyclic) {
+ /* MDMA already started */
+ if (chan->mchan.dir != DMA_MEM_TO_DEV &&
+ sg_dma_len(&sg_req->stm32_sgl_req) >
+ chan->mchan.sram_period)
+ reg->dma_scr |= STM32_DMA_SCR_DBM;
+ ret = stm32_dma_mdma_start(chan, sg_req);
+ if (ret < 0) {
chan->desc = NULL;
+ return;
}
- stm32_dma_start_transfer(chan);
}
}
+
+ chan->next_sg++;
+
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
+ stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg->dma_spar);
+ stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg->dma_sm0ar);
+ stm32_dma_write(dmadev, STM32_DMA_SFCR(chan->id), reg->dma_sfcr);
+ stm32_dma_write(dmadev, STM32_DMA_SM1AR(chan->id), reg->dma_sm1ar);
+ stm32_dma_write(dmadev, STM32_DMA_SNDTR(chan->id), reg->dma_sndtr);
+
+ if (chan->desc->cyclic)
+ stm32_dma_configure_next_sg(chan);
+
+ stm32_dma_dump_reg(chan);
+
+ /* Start DMA */
+ chan->busy = true;
+ reg->dma_scr |= STM32_DMA_SCR_EN;
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
+
+ dev_dbg(chan2dev(chan), "vchan %pK: started\n", &chan->vchan);
+}
+
+static void stm32_dma_handle_chan_done(struct stm32_dma_chan *chan)
+{
+ if (!chan->desc)
+ return;
+
+ if (chan->desc->cyclic) {
+ vchan_cyclic_callback(&chan->desc->vdesc);
+ if (chan->use_mdma)
+ return;
+ chan->next_sg++;
+ stm32_dma_configure_next_sg(chan);
+ } else {
+ chan->busy = false;
+ if (chan->use_mdma && chan->mchan.dir != DMA_MEM_TO_DEV)
+ return;
+ if (chan->next_sg == chan->desc->num_sgs) {
+ list_del(&chan->desc->vdesc.node);
+ vchan_cookie_complete(&chan->desc->vdesc);
+ chan->desc = NULL;
+ }
+
+ stm32_dma_start_transfer(chan);
+ }
}
static irqreturn_t stm32_dma_chan_irq(int irq, void *devid)
@@ -695,7 +1096,6 @@ static void stm32_dma_issue_pending(struct dma_chan *c)
if (vchan_issue_pending(&chan->vchan) && !chan->desc && !chan->busy) {
dev_dbg(chan2dev(chan), "vchan %pK: issued\n", &chan->vchan);
stm32_dma_start_transfer(chan);
-
}
spin_unlock_irqrestore(&chan->vchan.lock, flags);
}
@@ -836,16 +1236,128 @@ static void stm32_dma_clear_reg(struct stm32_dma_chan_reg *regs)
memset(regs, 0, sizeof(struct stm32_dma_chan_reg));
}
+static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
+ struct scatterlist *sgl, u32 sg_len,
+ struct stm32_dma_desc *desc)
+{
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct scatterlist *sg, *m_sg;
+ dma_addr_t dma_buf;
+ u32 len, num_sgs, sram_period;
+ int i, j, ret;
+
+ desc->dma_buf_cpu = gen_pool_dma_alloc(dmadev->sram_pool,
+ chan->sram_size,
+ &desc->dma_buf);
+ if (!desc->dma_buf_cpu)
+ return -ENOMEM;
+
+ sram_period = chan->sram_size / 2;
+
+ for_each_sg(sgl, sg, sg_len, i) {
+ struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[i].m_desc;
+
+ len = sg_dma_len(sg);
+ desc->sg_req[i].stm32_sgl_req = *sg;
+ num_sgs = 1;
+
+ if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ if (len > chan->sram_size) {
+ dev_err(chan2dev(chan),
+ "max buf size = %d bytes\n",
+ chan->sram_size);
+ goto free_alloc;
+ }
+ } else {
+ /*
+ * Build new sg for MDMA transfer
+ * Scatter DMA Req into several SDRAM transfer
+ */
+ if (len > sram_period)
+ num_sgs = len / sram_period;
+ }
+
+ ret = sg_alloc_table(&m_desc->sgt, num_sgs, GFP_ATOMIC);
+ if (ret) {
+ dev_err(chan2dev(chan), "MDMA sg table alloc failed\n");
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dma_buf = sg_dma_address(sg);
+ for_each_sg(m_desc->sgt.sgl, m_sg, num_sgs, j) {
+ size_t bytes = min_t(size_t, len, sram_period);
+
+ sg_dma_address(m_sg) = dma_buf;
+ sg_dma_len(m_sg) = bytes;
+ dma_buf += bytes;
+ len -= bytes;
+ }
+ }
+
+ chan->mchan.sram_buf = desc->dma_buf;
+ chan->mchan.sram_period = sram_period;
+ chan->mchan.num_sgs = num_sgs;
+
+ return 0;
+
+err:
+ for (j = 0; j < i; j++)
+ sg_free_table(&desc->sg_req[j].m_desc.sgt);
+free_alloc:
+ gen_pool_free(dmadev->sram_pool, (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ return ret;
+}
+
+static int stm32_dma_setup_sg_requests(struct stm32_dma_chan *chan,
+ struct scatterlist *sgl,
+ unsigned int sg_len,
+ enum dma_transfer_direction direction,
+ struct stm32_dma_desc *desc)
+{
+ struct scatterlist *sg;
+ u32 nb_data_items;
+ int i, ret;
+ enum dma_slave_buswidth buswidth;
+
+ for_each_sg(sgl, sg, sg_len, i) {
+ ret = stm32_dma_set_xfer_param(chan, direction, &buswidth,
+ sg_dma_len(sg));
+ if (ret < 0)
+ return ret;
+
+ nb_data_items = sg_dma_len(sg) / buswidth;
+ if (nb_data_items > STM32_DMA_ALIGNED_MAX_DATA_ITEMS) {
+ dev_err(chan2dev(chan), "nb items not supported\n");
+ return -EINVAL;
+ }
+
+ stm32_dma_clear_reg(&desc->sg_req[i].chan_reg);
+ desc->sg_req[i].chan_reg.dma_scr = chan->chan_reg.dma_scr;
+ desc->sg_req[i].chan_reg.dma_sfcr = chan->chan_reg.dma_sfcr;
+ desc->sg_req[i].chan_reg.dma_spar = chan->chan_reg.dma_spar;
+ desc->sg_req[i].chan_reg.dma_sm0ar = sg_dma_address(sg);
+ desc->sg_req[i].chan_reg.dma_sm1ar = sg_dma_address(sg);
+ if (chan->use_mdma)
+ desc->sg_req[i].chan_reg.dma_sm1ar +=
+ chan->mchan.sram_period;
+ desc->sg_req[i].chan_reg.dma_sndtr = nb_data_items;
+ }
+
+ desc->num_sgs = sg_len;
+
+ return 0;
+}
+
static struct dma_async_tx_descriptor *stm32_dma_prep_slave_sg(
struct dma_chan *c, struct scatterlist *sgl,
u32 sg_len, enum dma_transfer_direction direction,
unsigned long flags, void *context)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+
struct stm32_dma_desc *desc;
- struct scatterlist *sg;
- enum dma_slave_buswidth buswidth;
- u32 nb_data_items;
int i, ret;
if (!chan->config_init) {
@@ -868,48 +1380,141 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_slave_sg(
else
chan->chan_reg.dma_scr &= ~STM32_DMA_SCR_PFCTRL;
- for_each_sg(sgl, sg, sg_len, i) {
- ret = stm32_dma_set_xfer_param(chan, direction, &buswidth,
- sg_dma_len(sg));
- if (ret < 0)
- goto err;
-
- desc->sg_req[i].len = sg_dma_len(sg);
+ if (chan->use_mdma) {
+ struct sg_table new_sgt;
+ struct scatterlist *s, *_sgl;
- nb_data_items = desc->sg_req[i].len / buswidth;
- if (nb_data_items > STM32_DMA_ALIGNED_MAX_DATA_ITEMS) {
- dev_err(chan2dev(chan), "nb items not supported\n");
- goto err;
+ chan->mchan.dir = direction;
+ ret = stm32_dma_mdma_prep_slave_sg(chan, sgl, sg_len, desc);
+ if (ret < 0)
+ return NULL;
+
+ ret = sg_alloc_table(&new_sgt, sg_len, GFP_ATOMIC);
+ if (ret)
+ dev_err(chan2dev(chan), "DMA sg table alloc failed\n");
+
+ for_each_sg(new_sgt.sgl, s, sg_len, i) {
+ _sgl = sgl;
+ sg_dma_len(s) =
+ min(sg_dma_len(_sgl), chan->mchan.sram_period);
+ s->dma_address = chan->mchan.sram_buf;
+ _sgl = sg_next(_sgl);
}
- stm32_dma_clear_reg(&desc->sg_req[i].chan_reg);
- desc->sg_req[i].chan_reg.dma_scr = chan->chan_reg.dma_scr;
- desc->sg_req[i].chan_reg.dma_sfcr = chan->chan_reg.dma_sfcr;
- desc->sg_req[i].chan_reg.dma_spar = chan->chan_reg.dma_spar;
- desc->sg_req[i].chan_reg.dma_sm0ar = sg_dma_address(sg);
- desc->sg_req[i].chan_reg.dma_sm1ar = sg_dma_address(sg);
- desc->sg_req[i].chan_reg.dma_sndtr = nb_data_items;
+ ret = stm32_dma_setup_sg_requests(chan, new_sgt.sgl, sg_len,
+ direction, desc);
+ sg_free_table(&new_sgt);
+ if (ret < 0)
+ goto err;
+ } else {
+ /* Prepare a normal DMA transfer */
+ ret = stm32_dma_setup_sg_requests(chan, sgl, sg_len, direction,
+ desc);
+ if (ret < 0)
+ goto err;
}
- desc->num_sgs = sg_len;
desc->cyclic = false;
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
-
err:
+ if (chan->use_mdma) {
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+
+ for (i = 0; i < sg_len; i++)
+ sg_free_table(&desc->sg_req[i].m_desc.sgt);
+
+ gen_pool_free(dmadev->sram_pool,
+ (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ }
kfree(desc);
+
return NULL;
}
+static int stm32_dma_mdma_prep_dma_cyclic(struct stm32_dma_chan *chan,
+ dma_addr_t buf_addr, size_t buf_len,
+ size_t period_len,
+ struct stm32_dma_desc *desc)
+{
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+ struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[0].m_desc;
+ struct dma_slave_config config;
+ dma_addr_t mem;
+ int ret;
+
+ chan->sram_size = ALIGN(period_len, STM32_DMA_SRAM_GRANULARITY);
+ desc->dma_buf_cpu = gen_pool_dma_alloc(dmadev->sram_pool,
+ 2 * chan->sram_size,
+ &desc->dma_buf);
+ if (!desc->dma_buf_cpu)
+ return -ENOMEM;
+
+ memset(&config, 0, sizeof(config));
+ mem = buf_addr;
+
+ /* Configure MDMA channel */
+ if (chan->mchan.dir == DMA_MEM_TO_DEV)
+ config.dst_addr = desc->dma_buf;
+ else
+ config.src_addr = desc->dma_buf;
+ ret = dmaengine_slave_config(mchan->chan, &config);
+ if (ret < 0)
+ goto err;
+
+ /* Prepare MDMA descriptor */
+ m_desc->desc = dmaengine_prep_dma_cyclic(mchan->chan, buf_addr, buf_len,
+ period_len, chan->mchan.dir,
+ DMA_PREP_INTERRUPT);
+
+ if (!m_desc->desc) {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ ret = dma_submit_error(dmaengine_submit(m_desc->desc));
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "MDMA submit failed\n");
+ goto err;
+ }
+
+ dma_async_issue_pending(mchan->chan);
+
+ /*
+ * In case of M2D transfer, we have to generate dummy DMA transfer to
+ * copy 1 period of data into SRAM
+ */
+ if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ ret = stm32_dma_dummy_memcpy_xfer(chan);
+ if (ret < 0) {
+ dev_err(chan2dev(chan),
+ "stm32_dma_dummy_memcpy_xfer failed\n");
+ dmaengine_terminate_async(mchan->chan);
+ goto err;
+ }
+ }
+
+ return 0;
+err:
+ gen_pool_free(dmadev->sram_pool,
+ (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ return ret;
+}
+
static struct dma_async_tx_descriptor *stm32_dma_prep_dma_cyclic(
struct dma_chan *c, dma_addr_t buf_addr, size_t buf_len,
size_t period_len, enum dma_transfer_direction direction,
unsigned long flags)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_chan_reg *chan_reg = &chan->chan_reg;
struct stm32_dma_desc *desc;
enum dma_slave_buswidth buswidth;
u32 num_periods, nb_data_items;
+ dma_addr_t dma_buf = 0;
int i, ret;
if (!buf_len || !period_len) {
@@ -957,28 +1562,49 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_dma_cyclic(
/* Clear periph ctrl if client set it */
chan->chan_reg.dma_scr &= ~STM32_DMA_SCR_PFCTRL;
- num_periods = buf_len / period_len;
+ if (chan->use_mdma)
+ num_periods = 1;
+ else
+ num_periods = buf_len / period_len;
desc = stm32_dma_alloc_desc(num_periods);
if (!desc)
return NULL;
- for (i = 0; i < num_periods; i++) {
- desc->sg_req[i].len = period_len;
+ desc->num_sgs = num_periods;
+ desc->cyclic = true;
+ if (chan->use_mdma) {
+ chan->mchan.dir = direction;
+
+ ret = stm32_dma_mdma_prep_dma_cyclic(chan, buf_addr, buf_len,
+ period_len, desc);
+ if (ret < 0)
+ return NULL;
+ dma_buf = desc->dma_buf;
+ } else {
+ dma_buf = buf_addr;
+ }
+
+ for (i = 0; i < num_periods; i++) {
+ sg_dma_len(&desc->sg_req[i].stm32_sgl_req) = period_len;
+ sg_dma_address(&desc->sg_req[i].stm32_sgl_req) = dma_buf;
stm32_dma_clear_reg(&desc->sg_req[i].chan_reg);
- desc->sg_req[i].chan_reg.dma_scr = chan->chan_reg.dma_scr;
- desc->sg_req[i].chan_reg.dma_sfcr = chan->chan_reg.dma_sfcr;
- desc->sg_req[i].chan_reg.dma_spar = chan->chan_reg.dma_spar;
- desc->sg_req[i].chan_reg.dma_sm0ar = buf_addr;
- desc->sg_req[i].chan_reg.dma_sm1ar = buf_addr;
+ desc->sg_req[i].chan_reg.dma_scr = chan_reg->dma_scr;
+ desc->sg_req[i].chan_reg.dma_sfcr = chan_reg->dma_sfcr;
+ desc->sg_req[i].chan_reg.dma_spar = chan_reg->dma_spar;
+ if (chan->use_mdma) {
+ desc->sg_req[i].chan_reg.dma_sm0ar = desc->dma_buf;
+ desc->sg_req[i].chan_reg.dma_sm1ar = desc->dma_buf +
+ chan->sram_size;
+ } else {
+ desc->sg_req[i].chan_reg.dma_sm0ar = dma_buf;
+ desc->sg_req[i].chan_reg.dma_sm1ar = dma_buf;
+ dma_buf += period_len;
+ }
desc->sg_req[i].chan_reg.dma_sndtr = nb_data_items;
- buf_addr += period_len;
}
- desc->num_sgs = num_periods;
- desc->cyclic = true;
-
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
}
@@ -1019,13 +1645,13 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_dma_memcpy(
STM32_DMA_SCR_PINC |
STM32_DMA_SCR_TCIE |
STM32_DMA_SCR_TEIE;
- desc->sg_req[i].chan_reg.dma_sfcr |= STM32_DMA_SFCR_MASK;
+ desc->sg_req[i].chan_reg.dma_sfcr &= ~STM32_DMA_SFCR_MASK;
desc->sg_req[i].chan_reg.dma_sfcr |=
STM32_DMA_SFCR_FTH(threshold);
desc->sg_req[i].chan_reg.dma_spar = src + offset;
desc->sg_req[i].chan_reg.dma_sm0ar = dest + offset;
desc->sg_req[i].chan_reg.dma_sndtr = xfer_count;
- desc->sg_req[i].len = xfer_count;
+ sg_dma_len(&desc->sg_req[i].stm32_sgl_req) = xfer_count;
}
desc->num_sgs = num_sgs;
@@ -1034,18 +1660,6 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_dma_memcpy(
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
}
-static u32 stm32_dma_get_remaining_bytes(struct stm32_dma_chan *chan)
-{
- u32 dma_scr, width, ndtr;
- struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
-
- dma_scr = stm32_dma_read(dmadev, STM32_DMA_SCR(chan->id));
- width = STM32_DMA_SCR_PSIZE_GET(dma_scr);
- ndtr = stm32_dma_read(dmadev, STM32_DMA_SNDTR(chan->id));
-
- return ndtr << width;
-}
-
static size_t stm32_dma_desc_residue(struct stm32_dma_chan *chan,
struct stm32_dma_desc *desc,
u32 next_sg)
@@ -1054,6 +1668,10 @@ static size_t stm32_dma_desc_residue(struct stm32_dma_chan *chan,
u32 residue = 0;
int i;
+ /* Drain case */
+ if (chan->residue_after_drain)
+ return chan->residue_after_drain;
+
/*
* In cyclic mode, for the last period, residue = remaining bytes from
* NDTR
@@ -1069,7 +1687,7 @@ static size_t stm32_dma_desc_residue(struct stm32_dma_chan *chan,
* transferred
*/
for (i = next_sg; i < desc->num_sgs; i++)
- residue += desc->sg_req[i].len;
+ residue += sg_dma_len(&desc->sg_req[i].stm32_sgl_req);
residue += stm32_dma_get_remaining_bytes(chan);
end:
@@ -1089,11 +1707,23 @@ static enum dma_status stm32_dma_tx_status(struct dma_chan *c,
struct dma_tx_state *state)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
struct virt_dma_desc *vdesc;
enum dma_status status;
unsigned long flags;
u32 residue = 0;
+ /*
+ * When DMA/MDMA chain is used, we return the status of MDMA in cyclic
+ * mode and for D2M transfer in sg mode in order to return the correct
+ * residue if any
+ */
+ if (chan->desc && chan->use_mdma &&
+ (mchan->dir != DMA_MEM_TO_DEV || chan->desc->cyclic) &&
+ !chan->residue_after_drain)
+ return dmaengine_tx_status(mchan->chan, mchan->chan->cookie,
+ state);
+
status = dma_cookie_status(c, cookie, state);
if (status == DMA_COMPLETE || !state)
return status;
@@ -1155,21 +1785,34 @@ static void stm32_dma_free_chan_resources(struct dma_chan *c)
static void stm32_dma_desc_free(struct virt_dma_desc *vdesc)
{
- kfree(container_of(vdesc, struct stm32_dma_desc, vdesc));
+ struct stm32_dma_desc *desc = to_stm32_dma_desc(vdesc);
+ struct stm32_dma_chan *chan = to_stm32_dma_chan(vdesc->tx.chan);
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ int i;
+
+ if (chan->use_mdma) {
+ for (i = 0; i < desc->num_sgs; i++)
+ sg_free_table(&desc->sg_req[i].m_desc.sgt);
+
+ gen_pool_free(dmadev->sram_pool,
+ (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ }
+
+ kfree(desc);
}
static void stm32_dma_set_config(struct stm32_dma_chan *chan,
struct stm32_dma_cfg *cfg)
{
stm32_dma_clear_reg(&chan->chan_reg);
-
chan->chan_reg.dma_scr = cfg->stream_config & STM32_DMA_SCR_CFG_MASK;
chan->chan_reg.dma_scr |= STM32_DMA_SCR_REQ(cfg->request_line);
-
- /* Enable Interrupts */
chan->chan_reg.dma_scr |= STM32_DMA_SCR_TEIE | STM32_DMA_SCR_TCIE;
-
chan->threshold = STM32_DMA_THRESHOLD_FTR_GET(cfg->features);
+ chan->use_mdma = STM32_DMA_MDMA_CHAIN_FTR_GET(cfg->features);
+ chan->sram_size = (1 << STM32_DMA_MDMA_SRAM_SIZE_GET(cfg->features)) *
+ STM32_DMA_SRAM_GRANULARITY;
}
static struct dma_chan *stm32_dma_of_xlate(struct of_phandle_args *dma_spec,
@@ -1207,6 +1850,9 @@ static struct dma_chan *stm32_dma_of_xlate(struct of_phandle_args *dma_spec,
stm32_dma_set_config(chan, &cfg);
+ if (!dmadev->sram_pool || !chan->mchan.chan)
+ chan->use_mdma = 0;
+
return c;
}
@@ -1219,10 +1865,12 @@ MODULE_DEVICE_TABLE(of, stm32_dma_of_match);
static int stm32_dma_probe(struct platform_device *pdev)
{
struct stm32_dma_chan *chan;
+ struct stm32_dma_mdma *mchan;
struct stm32_dma_device *dmadev;
struct dma_device *dd;
const struct of_device_id *match;
struct resource *res;
+ char name[4];
int i, ret;
match = of_match_device(stm32_dma_of_match, &pdev->dev);
@@ -1258,6 +1906,13 @@ static int stm32_dma_probe(struct platform_device *pdev)
reset_control_deassert(dmadev->rst);
}
+ dmadev->sram_pool = of_gen_pool_get(pdev->dev.of_node, "sram", 0);
+ if (!dmadev->sram_pool)
+ dev_info(&pdev->dev, "no dma pool: can't use MDMA: %d\n", ret);
+ else
+ dev_dbg(&pdev->dev, "SRAM pool: %zu KiB\n",
+ gen_pool_size(dmadev->sram_pool) / 1024);
+
dma_cap_set(DMA_SLAVE, dd->cap_mask);
dma_cap_set(DMA_PRIVATE, dd->cap_mask);
dma_cap_set(DMA_CYCLIC, dd->cap_mask);
@@ -1293,6 +1948,16 @@ static int stm32_dma_probe(struct platform_device *pdev)
chan->id = i;
chan->vchan.desc_free = stm32_dma_desc_free;
vchan_init(&chan->vchan, dd);
+
+ mchan = &chan->mchan;
+ if (dmadev->sram_pool) {
+ snprintf(name, sizeof(name), "ch%d", chan->id);
+ mchan->chan = dma_request_slave_channel(dd->dev, name);
+ if (!mchan->chan)
+ dev_info(&pdev->dev,
+ "can't request MDMA chan for %s\n",
+ name);
+ }
}
ret = dma_async_device_register(dd);
@@ -1350,4 +2015,4 @@ static int __init stm32_dma_init(void)
{
return platform_driver_probe(&stm32_dma_driver, stm32_dma_probe);
}
-subsys_initcall(stm32_dma_init);
+device_initcall(stm32_dma_init);
^ permalink raw reply related
* [v2,5/7] dmaengine: stm32-mdma: Add DMA/MDMA chaining support
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
This patch adds support for M2M transfer triggered by STM32 DMA in order to
transfer data from/to SRAM to/from DDR.
Normally, this mode should not be needed as transferring data from/to DDR
is supported by the STM32 DMA.
However, the STM32 DMA don't have the ability to generate burst transfer
on the DDR as it only embeds only a 4-word FIFO although the minimal burst
length on the DDR is 8 words.
Due to this constraint, the STM32 DMA transfers data from/to DDR in a
single way and could lead to pollute the DDR.
To avoid this, we have to use SRAM for all transfers where STM32 DMA is
involved.
So, we need to add an intermediate M2M transfer handled by the MDMA, which
has the ability to generate burst transfer on the DDR, to copy data
from/to SRAM to/from DDR as described below:
For M2D: DDR --> MDMA --> SRAM --> DMA --> IP
For D2M: IP --> DMA --> SRAM --> MDMA --> DDR
This intermediate transfer is triggered by the STM32 DMA when his transfer
complete flag is set. In that way, we are able to build a DMA/MDMA
chaining transfer completely handled by HW.
This patch clearly adds support for M2M transfer triggered by HW.
This mode is not really available in dmaengine framework as normally M2M
transfers are triggered by SW.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
drivers/dma/stm32-mdma.c | 131 +++++++++++++++++++++++++++++++++++++++++------
1 file changed, 114 insertions(+), 17 deletions(-)
diff --git a/drivers/dma/stm32-mdma.c b/drivers/dma/stm32-mdma.c
index 06dd172..6b6e63b 100644
--- a/drivers/dma/stm32-mdma.c
+++ b/drivers/dma/stm32-mdma.c
@@ -211,6 +211,8 @@
#define STM32_MDMA_MAX_BURST 128
#define STM32_MDMA_VERY_HIGH_PRIORITY 0x11
+#define STM32_DMA_SRAM_GRANULARITY PAGE_SIZE
+
enum stm32_mdma_trigger_mode {
STM32_MDMA_BUFFER,
STM32_MDMA_BLOCK,
@@ -237,6 +239,7 @@ struct stm32_mdma_chan_config {
u32 transfer_config;
u32 mask_addr;
u32 mask_data;
+ bool m2m_hw;
};
struct stm32_mdma_hwdesc {
@@ -262,6 +265,7 @@ struct stm32_mdma_desc {
u32 ccr;
bool cyclic;
u32 count;
+ enum dma_transfer_direction dir;
struct stm32_mdma_desc_node node[];
};
@@ -577,13 +581,25 @@ static int stm32_mdma_set_xfer_param(struct stm32_mdma_chan *chan,
dst_addr = chan->dma_config.dst_addr;
/* Set device data size */
+ if (chan_config->m2m_hw)
+ dst_addr_width =
+ stm32_mdma_get_max_width(dst_addr, buf_len,
+ STM32_MDMA_MAX_BUF_LEN);
+
dst_bus_width = stm32_mdma_get_width(chan, dst_addr_width);
if (dst_bus_width < 0)
return dst_bus_width;
ctcr &= ~STM32_MDMA_CTCR_DSIZE_MASK;
ctcr |= STM32_MDMA_CTCR_DSIZE(dst_bus_width);
+ if (chan_config->m2m_hw) {
+ ctcr &= ~STM32_MDMA_CTCR_DINCOS_MASK;
+ ctcr |= STM32_MDMA_CTCR_DINCOS(dst_bus_width);
+ }
/* Set device burst value */
+ if (chan_config->m2m_hw)
+ dst_maxburst = STM32_MDMA_MAX_BUF_LEN / dst_addr_width;
+
dst_best_burst = stm32_mdma_get_best_burst(buf_len, tlen,
dst_maxburst,
dst_addr_width);
@@ -626,13 +642,25 @@ static int stm32_mdma_set_xfer_param(struct stm32_mdma_chan *chan,
src_addr = chan->dma_config.src_addr;
/* Set device data size */
+ if (chan_config->m2m_hw)
+ src_addr_width =
+ stm32_mdma_get_max_width(src_addr, buf_len,
+ STM32_MDMA_MAX_BUF_LEN);
+
src_bus_width = stm32_mdma_get_width(chan, src_addr_width);
if (src_bus_width < 0)
return src_bus_width;
ctcr &= ~STM32_MDMA_CTCR_SSIZE_MASK;
ctcr |= STM32_MDMA_CTCR_SSIZE(src_bus_width);
+ if (chan_config->m2m_hw) {
+ ctcr &= ~STM32_MDMA_CTCR_SINCOS_MASK;
+ ctcr |= STM32_MDMA_CTCR_SINCOS(src_bus_width);
+ }
/* Set device burst value */
+ if (chan_config->m2m_hw)
+ src_maxburst = STM32_MDMA_MAX_BUF_LEN / src_addr_width;
+
src_best_burst = stm32_mdma_get_best_burst(buf_len, tlen,
src_maxburst,
src_addr_width);
@@ -740,6 +768,7 @@ static int stm32_mdma_setup_xfer(struct stm32_mdma_chan *chan,
{
struct stm32_mdma_device *dmadev = stm32_mdma_get_dev(chan);
struct dma_slave_config *dma_config = &chan->dma_config;
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct scatterlist *sg;
dma_addr_t src_addr, dst_addr;
u32 ccr, ctcr, ctbr;
@@ -762,6 +791,8 @@ static int stm32_mdma_setup_xfer(struct stm32_mdma_chan *chan,
} else {
src_addr = dma_config->src_addr;
dst_addr = sg_dma_address(sg);
+ if (chan_config->m2m_hw)
+ src_addr += ((i & 1) ? sg_dma_len(sg) : 0);
ret = stm32_mdma_set_xfer_param(chan, direction, &ccr,
&ctcr, &ctbr, dst_addr,
sg_dma_len(sg));
@@ -780,8 +811,6 @@ static int stm32_mdma_setup_xfer(struct stm32_mdma_chan *chan,
/* Enable interrupts */
ccr &= ~STM32_MDMA_CCR_IRQ_MASK;
ccr |= STM32_MDMA_CCR_TEIE | STM32_MDMA_CCR_CTCIE;
- if (sg_len > 1)
- ccr |= STM32_MDMA_CCR_BTIE;
desc->ccr = ccr;
return 0;
@@ -793,7 +822,9 @@ stm32_mdma_prep_slave_sg(struct dma_chan *c, struct scatterlist *sgl,
unsigned long flags, void *context)
{
struct stm32_mdma_chan *chan = to_stm32_mdma_chan(c);
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct stm32_mdma_desc *desc;
+ struct stm32_mdma_hwdesc *hwdesc;
int i, ret;
/*
@@ -815,6 +846,20 @@ stm32_mdma_prep_slave_sg(struct dma_chan *c, struct scatterlist *sgl,
if (ret < 0)
goto xfer_setup_err;
+ /*
+ * In case of M2M HW transfer triggered by STM32 DMA, we do not have to
+ * clear the transfer complete flag by hardware in order to let the
+ * CPU rearm the DMA with the next sg element and update some data in
+ * dmaengine framework
+ */
+ if (chan_config->m2m_hw && direction == DMA_MEM_TO_DEV) {
+ for (i = 0; i < sg_len; i++) {
+ hwdesc = desc->node[i].hwdesc;
+ hwdesc->cmar = 0;
+ hwdesc->cmdr = 0;
+ }
+ }
+
desc->cyclic = false;
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
@@ -836,9 +881,10 @@ stm32_mdma_prep_dma_cyclic(struct dma_chan *c, dma_addr_t buf_addr,
struct stm32_mdma_chan *chan = to_stm32_mdma_chan(c);
struct stm32_mdma_device *dmadev = stm32_mdma_get_dev(chan);
struct dma_slave_config *dma_config = &chan->dma_config;
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct stm32_mdma_desc *desc;
dma_addr_t src_addr, dst_addr;
- u32 ccr, ctcr, ctbr, count;
+ u32 ccr, ctcr, ctbr, count, offset;
int i, ret;
/*
@@ -892,12 +938,29 @@ stm32_mdma_prep_dma_cyclic(struct dma_chan *c, dma_addr_t buf_addr,
desc->ccr = ccr;
/* Configure hwdesc list */
+ offset = ALIGN(period_len, STM32_DMA_SRAM_GRANULARITY);
for (i = 0; i < count; i++) {
if (direction == DMA_MEM_TO_DEV) {
+ /*
+ * When the DMA is configured in double buffer mode,
+ * the MDMA has to use 2 destination buffers to be
+ * compliant with this mode.
+ */
+ if (chan_config->m2m_hw && count > 1 && i % 2)
+ dst_addr = dma_config->dst_addr + offset;
+ else
+ dst_addr = dma_config->dst_addr;
src_addr = buf_addr + i * period_len;
- dst_addr = dma_config->dst_addr;
} else {
- src_addr = dma_config->src_addr;
+ /*
+ * When the DMA is configured in double buffer mode,
+ * the MDMA has to use 2 destination buffers to be
+ * compliant with this mode.
+ */
+ if (chan_config->m2m_hw && count > 1 && i % 2)
+ src_addr = dma_config->src_addr + offset;
+ else
+ src_addr = dma_config->src_addr;
dst_addr = buf_addr + i * period_len;
}
@@ -907,6 +970,7 @@ stm32_mdma_prep_dma_cyclic(struct dma_chan *c, dma_addr_t buf_addr,
}
desc->cyclic = true;
+ desc->dir = direction;
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
@@ -1287,14 +1351,28 @@ static size_t stm32_mdma_desc_residue(struct stm32_mdma_chan *chan,
{
struct stm32_mdma_device *dmadev = stm32_mdma_get_dev(chan);
struct stm32_mdma_hwdesc *hwdesc = desc->node[0].hwdesc;
- u32 cbndtr, residue, modulo, burst_size;
+ u32 residue = 0;
+ u32 modulo, burst_size;
+ dma_addr_t next_clar;
+ u32 cbndtr;
int i;
- residue = 0;
- for (i = curr_hwdesc + 1; i < desc->count; i++) {
+ /*
+ * Get the residue of pending descriptors
+ */
+ /* Get the next hw descriptor to process from current transfer */
+ next_clar = stm32_mdma_read(dmadev, STM32_MDMA_CLAR(chan->id));
+ for (i = desc->count - 1; i >= 0; i--) {
hwdesc = desc->node[i].hwdesc;
+
+ if (hwdesc->clar == next_clar)
+ break;/* Current transfer found, stop cumulating */
+
+ /* Cumulate residue of unprocessed hw descriptors */
residue += STM32_MDMA_CBNDTR_BNDT(hwdesc->cbndtr);
}
+
+ /* Read & cumulate the residue of the current transfer */
cbndtr = stm32_mdma_read(dmadev, STM32_MDMA_CBNDTR(chan->id));
residue += cbndtr & STM32_MDMA_CBNDTR_BNDT_MASK;
@@ -1314,24 +1392,39 @@ static enum dma_status stm32_mdma_tx_status(struct dma_chan *c,
struct dma_tx_state *state)
{
struct stm32_mdma_chan *chan = to_stm32_mdma_chan(c);
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct virt_dma_desc *vdesc;
enum dma_status status;
unsigned long flags;
u32 residue = 0;
status = dma_cookie_status(c, cookie, state);
- if ((status == DMA_COMPLETE) || (!state))
+ if (status == DMA_COMPLETE || !state)
return status;
spin_lock_irqsave(&chan->vchan.lock, flags);
vdesc = vchan_find_desc(&chan->vchan, cookie);
- if (chan->desc && cookie == chan->desc->vdesc.tx.cookie)
- residue = stm32_mdma_desc_residue(chan, chan->desc,
- chan->curr_hwdesc);
- else if (vdesc)
+ if (chan->desc && cookie == chan->desc->vdesc.tx.cookie) {
+ /*
+ * In case of M2D transfer triggered by STM32 DMA, the MDMA has
+ * always one period in advance in cyclic mode. So, we have to
+ * add 1 period of data to return the good residue to the
+ * client
+ */
+ if (chan_config->m2m_hw && chan->desc->dir == DMA_MEM_TO_DEV &&
+ chan->curr_hwdesc > 1)
+ residue =
+ stm32_mdma_desc_residue(chan, chan->desc,
+ chan->curr_hwdesc - 1);
+ else
+ residue = stm32_mdma_desc_residue(chan, chan->desc,
+ chan->curr_hwdesc);
+ } else if (vdesc) {
residue = stm32_mdma_desc_residue(chan,
to_stm32_mdma_desc(vdesc), 0);
+ }
+
dma_set_residue(state, residue);
spin_unlock_irqrestore(&chan->vchan.lock, flags);
@@ -1498,7 +1591,7 @@ static struct dma_chan *stm32_mdma_of_xlate(struct of_phandle_args *dma_spec,
struct dma_chan *c;
struct stm32_mdma_chan_config config;
- if (dma_spec->args_count < 5) {
+ if (dma_spec->args_count < 6) {
dev_err(mdma2dev(dmadev), "Bad number of args\n");
return NULL;
}
@@ -1508,6 +1601,7 @@ static struct dma_chan *stm32_mdma_of_xlate(struct of_phandle_args *dma_spec,
config.transfer_config = dma_spec->args[2];
config.mask_addr = dma_spec->args[3];
config.mask_data = dma_spec->args[4];
+ config.m2m_hw = dma_spec->args[5];
if (config.request >= dmadev->nr_requests) {
dev_err(mdma2dev(dmadev), "Bad request line\n");
@@ -1646,19 +1740,20 @@ static int stm32_mdma_probe(struct platform_device *pdev)
dmadev->irq = platform_get_irq(pdev, 0);
if (dmadev->irq < 0) {
dev_err(&pdev->dev, "failed to get IRQ\n");
- return dmadev->irq;
+ ret = dmadev->irq;
+ goto clk_free;
}
ret = devm_request_irq(&pdev->dev, dmadev->irq, stm32_mdma_irq_handler,
0, dev_name(&pdev->dev), dmadev);
if (ret) {
dev_err(&pdev->dev, "failed to request IRQ\n");
- return ret;
+ goto clk_free;
}
ret = dma_async_device_register(dd);
if (ret)
- return ret;
+ goto clk_free;
ret = of_dma_controller_register(of_node, stm32_mdma_of_xlate, dmadev);
if (ret < 0) {
@@ -1675,6 +1770,8 @@ static int stm32_mdma_probe(struct platform_device *pdev)
err_unregister:
dma_async_device_unregister(dd);
+clk_free:
+ clk_disable_unprepare(dmadev->clk);
return ret;
}
^ permalink raw reply related
* [v2,6/7] dmaengine: stm32-dma: enable descriptor_reuse
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
Enable client to resubmit already processed descriptors
in order to save descriptor creation time.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
drivers/dma/stm32-dma.c | 84 +++++++++++++++++++++++++++++++------------------
1 file changed, 54 insertions(+), 30 deletions(-)
diff --git a/drivers/dma/stm32-dma.c b/drivers/dma/stm32-dma.c
index 1571f2f..fac44ed 100644
--- a/drivers/dma/stm32-dma.c
+++ b/drivers/dma/stm32-dma.c
@@ -836,34 +836,8 @@ static int stm32_dma_mdma_start(struct stm32_dma_chan *chan,
{
struct stm32_dma_mdma *mchan = &chan->mchan;
struct stm32_dma_mdma_desc *m_desc = &sg_req->m_desc;
- struct dma_slave_config config;
int ret;
- /* Configure MDMA channel */
- memset(&config, 0, sizeof(config));
- if (mchan->dir == DMA_MEM_TO_DEV)
- config.dst_addr = mchan->sram_buf;
- else
- config.src_addr = mchan->sram_buf;
-
- ret = dmaengine_slave_config(mchan->chan, &config);
- if (ret < 0)
- goto error;
-
- /* Prepare MDMA descriptor */
- m_desc->desc = dmaengine_prep_slave_sg(mchan->chan, m_desc->sgt.sgl,
- m_desc->sgt.nents, mchan->dir,
- DMA_PREP_INTERRUPT);
- if (!m_desc->desc) {
- ret = -EINVAL;
- goto error;
- }
-
- if (mchan->dir != DMA_MEM_TO_DEV) {
- m_desc->desc->callback_result = stm32_mdma_chan_complete;
- m_desc->desc->callback_param = chan;
- }
-
ret = dma_submit_error(dmaengine_submit(m_desc->desc));
if (ret < 0) {
dev_err(chan2dev(chan), "MDMA submit failed\n");
@@ -1001,6 +975,7 @@ static void stm32_dma_start_transfer(struct stm32_dma_chan *chan)
chan->next_sg++;
+ reg->dma_scr &= ~STM32_DMA_SCR_EN;
stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg->dma_spar);
stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg->dma_sm0ar);
@@ -1238,9 +1213,11 @@ static void stm32_dma_clear_reg(struct stm32_dma_chan_reg *regs)
static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
struct scatterlist *sgl, u32 sg_len,
- struct stm32_dma_desc *desc)
+ struct stm32_dma_desc *desc,
+ unsigned long flags)
{
struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
struct scatterlist *sg, *m_sg;
dma_addr_t dma_buf;
u32 len, num_sgs, sram_period;
@@ -1256,12 +1233,13 @@ static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
for_each_sg(sgl, sg, sg_len, i) {
struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[i].m_desc;
+ struct dma_slave_config config;
len = sg_dma_len(sg);
desc->sg_req[i].stm32_sgl_req = *sg;
num_sgs = 1;
- if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ if (mchan->dir == DMA_MEM_TO_DEV) {
if (len > chan->sram_size) {
dev_err(chan2dev(chan),
"max buf size = %d bytes\n",
@@ -1293,6 +1271,38 @@ static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
dma_buf += bytes;
len -= bytes;
}
+
+ /* Configure MDMA channel */
+ memset(&config, 0, sizeof(config));
+ if (mchan->dir == DMA_MEM_TO_DEV)
+ config.dst_addr = desc->dma_buf;
+ else
+ config.src_addr = desc->dma_buf;
+
+ ret = dmaengine_slave_config(mchan->chan, &config);
+ if (ret < 0)
+ goto err;
+
+ /* Prepare MDMA descriptor */
+ m_desc->desc = dmaengine_prep_slave_sg(mchan->chan,
+ m_desc->sgt.sgl,
+ m_desc->sgt.nents,
+ mchan->dir,
+ DMA_PREP_INTERRUPT);
+
+ if (!m_desc->desc) {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ if (flags & DMA_CTRL_REUSE)
+ dmaengine_desc_set_reuse(m_desc->desc);
+
+ if (mchan->dir != DMA_MEM_TO_DEV) {
+ m_desc->desc->callback_result =
+ stm32_mdma_chan_complete;
+ m_desc->desc->callback_param = chan;
+ }
}
chan->mchan.sram_buf = desc->dma_buf;
@@ -1302,8 +1312,12 @@ static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
return 0;
err:
- for (j = 0; j < i; j++)
+ for (j = 0; j < i; j++) {
+ struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[j].m_desc;
+
+ m_desc->desc = NULL;
sg_free_table(&desc->sg_req[j].m_desc.sgt);
+ }
free_alloc:
gen_pool_free(dmadev->sram_pool, (unsigned long)desc->dma_buf_cpu,
chan->sram_size);
@@ -1385,7 +1399,8 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_slave_sg(
struct scatterlist *s, *_sgl;
chan->mchan.dir = direction;
- ret = stm32_dma_mdma_prep_slave_sg(chan, sgl, sg_len, desc);
+ ret = stm32_dma_mdma_prep_slave_sg(chan, sgl, sg_len, desc,
+ flags);
if (ret < 0)
return NULL;
@@ -1791,6 +1806,14 @@ static void stm32_dma_desc_free(struct virt_dma_desc *vdesc)
int i;
if (chan->use_mdma) {
+ struct stm32_dma_mdma_desc *m_desc;
+
+ for (i = 0; i < desc->num_sgs; i++) {
+ m_desc = &desc->sg_req[i].m_desc;
+ dmaengine_desc_free(m_desc->desc);
+ m_desc->desc = NULL;
+ }
+
for (i = 0; i < desc->num_sgs; i++)
sg_free_table(&desc->sg_req[i].m_desc.sgt);
@@ -1934,6 +1957,7 @@ static int stm32_dma_probe(struct platform_device *pdev)
dd->directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
dd->residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
dd->max_burst = STM32_DMA_MAX_BURST;
+ dd->descriptor_reuse = true;
dd->dev = &pdev->dev;
INIT_LIST_HEAD(&dd->channels);
^ permalink raw reply related
* [v2,7/7] dmaengine: stm32-mdma: enable descriptor_reuse
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
enable reuse to spare descriptors creation on critical UC.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
drivers/dma/stm32-mdma.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/dma/stm32-mdma.c b/drivers/dma/stm32-mdma.c
index 6b6e63b..80a17dd 100644
--- a/drivers/dma/stm32-mdma.c
+++ b/drivers/dma/stm32-mdma.c
@@ -1715,6 +1715,8 @@ static int stm32_mdma_probe(struct platform_device *pdev)
dd->device_resume = stm32_mdma_resume;
dd->device_terminate_all = stm32_mdma_terminate_all;
dd->device_synchronize = stm32_mdma_synchronize;
+ dd->descriptor_reuse = true;
+
dd->src_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) |
BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) |
BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) |
^ permalink raw reply related
* [v2,4/7] dmaengine: stm32-dma: Add DMA/MDMA chaining support
From: kbuild test robot @ 2018-09-28 12:36 UTC (permalink / raw)
To: Pierre-Yves MORDRET
Cc: kbuild-all, Vinod Koul, Rob Herring, Mark Rutland,
Alexandre Torgue, Maxime Coquelin, Dan Williams, devicetree,
dmaengine, linux-arm-kernel, linux-kernel
Hi Pierre-Yves,
I love your patch! Perhaps something to improve:
[auto build test WARNING on robh/for-next]
[also build test WARNING on v4.19-rc5 next-20180928]
[if your patch is applied to the wrong git tree, please drop us a note to help improve the system]
url: https://github.com/0day-ci/linux/commits/Pierre-Yves-MORDRET/Add-DMA-MDMA-chaining-support/20180928-164058
base: https://git.kernel.org/pub/scm/linux/kernel/git/robh/linux.git for-next
config: i386-allyesconfig (attached as .config)
compiler: gcc-7 (Debian 7.3.0-1) 7.3.0
reproduce:
# save the attached .config to linux build tree
make ARCH=i386
Note: it may well be a FALSE warning. FWIW you are at least aware of it now.
http://gcc.gnu.org/wiki/Better_Uninitialized_Warnings
All warnings (new ones prefixed by >>):
drivers//dma/stm32-dma.c: In function 'stm32_dma_probe':
>> drivers//dma/stm32-dma.c:1911:3: warning: 'ret' may be used uninitialized in this function [-Wmaybe-uninitialized]
dev_info(&pdev->dev, "no dma pool: can't use MDMA: %d\n", ret);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
vim +/ret +1911 drivers//dma/stm32-dma.c
1864
1865 static int stm32_dma_probe(struct platform_device *pdev)
1866 {
1867 struct stm32_dma_chan *chan;
1868 struct stm32_dma_mdma *mchan;
1869 struct stm32_dma_device *dmadev;
1870 struct dma_device *dd;
1871 const struct of_device_id *match;
1872 struct resource *res;
1873 char name[4];
1874 int i, ret;
1875
1876 match = of_match_device(stm32_dma_of_match, &pdev->dev);
1877 if (!match) {
1878 dev_err(&pdev->dev, "Error: No device match found\n");
1879 return -ENODEV;
1880 }
1881
1882 dmadev = devm_kzalloc(&pdev->dev, sizeof(*dmadev), GFP_KERNEL);
1883 if (!dmadev)
1884 return -ENOMEM;
1885
1886 dd = &dmadev->ddev;
1887
1888 res = platform_get_resource(pdev, IORESOURCE_MEM, 0);
1889 dmadev->base = devm_ioremap_resource(&pdev->dev, res);
1890 if (IS_ERR(dmadev->base))
1891 return PTR_ERR(dmadev->base);
1892
1893 dmadev->clk = devm_clk_get(&pdev->dev, NULL);
1894 if (IS_ERR(dmadev->clk)) {
1895 dev_err(&pdev->dev, "Error: Missing controller clock\n");
1896 return PTR_ERR(dmadev->clk);
1897 }
1898
1899 dmadev->mem2mem = of_property_read_bool(pdev->dev.of_node,
1900 "st,mem2mem");
1901
1902 dmadev->rst = devm_reset_control_get(&pdev->dev, NULL);
1903 if (!IS_ERR(dmadev->rst)) {
1904 reset_control_assert(dmadev->rst);
1905 udelay(2);
1906 reset_control_deassert(dmadev->rst);
1907 }
1908
1909 dmadev->sram_pool = of_gen_pool_get(pdev->dev.of_node, "sram", 0);
1910 if (!dmadev->sram_pool)
> 1911 dev_info(&pdev->dev, "no dma pool: can't use MDMA: %d\n", ret);
1912 else
1913 dev_dbg(&pdev->dev, "SRAM pool: %zu KiB\n",
1914 gen_pool_size(dmadev->sram_pool) / 1024);
1915
1916 dma_cap_set(DMA_SLAVE, dd->cap_mask);
1917 dma_cap_set(DMA_PRIVATE, dd->cap_mask);
1918 dma_cap_set(DMA_CYCLIC, dd->cap_mask);
1919 dd->device_alloc_chan_resources = stm32_dma_alloc_chan_resources;
1920 dd->device_free_chan_resources = stm32_dma_free_chan_resources;
1921 dd->device_tx_status = stm32_dma_tx_status;
1922 dd->device_issue_pending = stm32_dma_issue_pending;
1923 dd->device_prep_slave_sg = stm32_dma_prep_slave_sg;
1924 dd->device_prep_dma_cyclic = stm32_dma_prep_dma_cyclic;
1925 dd->device_config = stm32_dma_slave_config;
1926 dd->device_terminate_all = stm32_dma_terminate_all;
1927 dd->device_synchronize = stm32_dma_synchronize;
1928 dd->src_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) |
1929 BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) |
1930 BIT(DMA_SLAVE_BUSWIDTH_4_BYTES);
1931 dd->dst_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) |
1932 BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) |
1933 BIT(DMA_SLAVE_BUSWIDTH_4_BYTES);
1934 dd->directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
1935 dd->residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
1936 dd->max_burst = STM32_DMA_MAX_BURST;
1937 dd->dev = &pdev->dev;
1938 INIT_LIST_HEAD(&dd->channels);
1939
1940 if (dmadev->mem2mem) {
1941 dma_cap_set(DMA_MEMCPY, dd->cap_mask);
1942 dd->device_prep_dma_memcpy = stm32_dma_prep_dma_memcpy;
1943 dd->directions |= BIT(DMA_MEM_TO_MEM);
1944 }
1945
1946 for (i = 0; i < STM32_DMA_MAX_CHANNELS; i++) {
1947 chan = &dmadev->chan[i];
1948 chan->id = i;
1949 chan->vchan.desc_free = stm32_dma_desc_free;
1950 vchan_init(&chan->vchan, dd);
1951
1952 mchan = &chan->mchan;
1953 if (dmadev->sram_pool) {
1954 snprintf(name, sizeof(name), "ch%d", chan->id);
1955 mchan->chan = dma_request_slave_channel(dd->dev, name);
1956 if (!mchan->chan)
1957 dev_info(&pdev->dev,
1958 "can't request MDMA chan for %s\n",
1959 name);
1960 }
1961 }
1962
1963 ret = dma_async_device_register(dd);
1964 if (ret)
1965 return ret;
1966
1967 for (i = 0; i < STM32_DMA_MAX_CHANNELS; i++) {
1968 chan = &dmadev->chan[i];
1969 res = platform_get_resource(pdev, IORESOURCE_IRQ, i);
1970 if (!res) {
1971 ret = -EINVAL;
1972 dev_err(&pdev->dev, "No irq resource for chan %d\n", i);
1973 goto err_unregister;
1974 }
1975 chan->irq = res->start;
1976 ret = devm_request_irq(&pdev->dev, chan->irq,
1977 stm32_dma_chan_irq, 0,
1978 dev_name(chan2dev(chan)), chan);
1979 if (ret) {
1980 dev_err(&pdev->dev,
1981 "request_irq failed with err %d channel %d\n",
1982 ret, i);
1983 goto err_unregister;
1984 }
1985 }
1986
1987 ret = of_dma_controller_register(pdev->dev.of_node,
1988 stm32_dma_of_xlate, dmadev);
1989 if (ret < 0) {
1990 dev_err(&pdev->dev,
1991 "STM32 DMA DMA OF registration failed %d\n", ret);
1992 goto err_unregister;
1993 }
1994
1995 platform_set_drvdata(pdev, dmadev);
1996
1997 dev_info(&pdev->dev, "STM32 DMA driver registered\n");
1998
1999 return 0;
2000
2001 err_unregister:
2002 dma_async_device_unregister(dd);
2003
2004 return ret;
2005 }
2006
---
0-DAY kernel test infrastructure Open Source Technology Center
https://lists.01.org/pipermail/kbuild-all Intel Corporation
^ permalink raw reply
* [v3,1/7] dt-bindings: stm32-dma: Add DMA/MDMA chaining support bindings
From: Pierre Yves MORDRET @ 2018-09-28 13:01 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
This patch adds dma bindings to support DMA/MDMA chaining transfer.
1 bit is to manage both DMA FIFO Threshold
1 bit is to manage DMA/MDMA Chaining features.
2 bits are used to specify SDRAM size to use for DMA/MDMA chaining.
The size in bytes of a certain order is given by the formula:
(2 ^ order) * PAGE_SIZE.
The order is given by those 2 bits.
For cyclic, whether chaining is chosen, any value above 1 can be set :
SRAM buffer size will rely on period size and not on this DT value.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v3:
v2:
* rework content
v1:
* Initial
---
---
.../devicetree/bindings/dma/stm32-dma.txt | 27 +++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/Documentation/devicetree/bindings/dma/stm32-dma.txt b/Documentation/devicetree/bindings/dma/stm32-dma.txt
index c5f5190..2bac8c7 100644
--- a/Documentation/devicetree/bindings/dma/stm32-dma.txt
+++ b/Documentation/devicetree/bindings/dma/stm32-dma.txt
@@ -17,6 +17,12 @@ Optional properties:
- resets: Reference to a reset controller asserting the DMA controller
- st,mem2mem: boolean; if defined, it indicates that the controller supports
memory-to-memory transfer
+- dmas: A list of eight dma specifiers, one for each entry in dma-names.
+ Refer to stm32-mdma.txt for more details.
+- dma-names: should contain "ch0", "ch1", "ch2", "ch3", "ch4", "ch5", "ch6" and
+ "ch7" and represents each STM32 DMA channel connected to a STM32 MDMA one.
+- memory-region : phandle to a node describing memory to be used for
+ M2M intermediate transfer between DMA and MDMA.
Example:
@@ -36,6 +42,16 @@ Example:
st,mem2mem;
resets = <&rcc 150>;
dma-requests = <8>;
+ dmas = <&mdma1 8 0x10 0x1200000a 0x40026408 0x00000020 1>,
+ <&mdma1 9 0x10 0x1200000a 0x40026408 0x00000800 1>,
+ <&mdma1 10 0x10 0x1200000a 0x40026408 0x00200000 1>,
+ <&mdma1 11 0x10 0x1200000a 0x40026408 0x08000000 1>,
+ <&mdma1 12 0x10 0x1200000a 0x4002640C 0x00000020 1>,
+ <&mdma1 13 0x10 0x1200000a 0x4002640C 0x00000800 1>,
+ <&mdma1 14 0x10 0x1200000a 0x4002640C 0x00200000 1>,
+ <&mdma1 15 0x10 0x1200000a 0x4002640C 0x08000000 1>;
+ dma-names = "ch0", "ch1", "ch2", "ch3", "ch4", "ch5", "ch6", "ch7";
+ memory-region = <&sram_dmapool>;
};
* DMA client
@@ -68,7 +84,16 @@ channel: a phandle to the DMA controller plus the following four integer cells:
0x1: 1/2 full FIFO
0x2: 3/4 full FIFO
0x3: full FIFO
-
+ -bit 2: Intermediate M2M transfer from/to DDR to/from SRAM throughout MDMA
+ 0: MDMA not used to generate an intermediate M2M transfer
+ 1: MDMA used to generate an intermediate M2M transfer.
+ -bit 3-4: indicated SRAM Buffer size in (2^order)*PAGE_SIZE.
+ PAGE_SIZE is given by Linux at 4KiB: include/asm-generic/page.h.
+ Order is given by those 2 bits starting at 0.
+ Valid only whether Intermediate M2M transfer is set.
+ For cyclic, whether Intermediate M2M transfer is chosen, any value can
+ be set: SRAM buffer size will rely on period size and not on this DT
+ value.
Example:
^ permalink raw reply related
* [v3,2/7] dt-bindings: stm32-dmamux: Add one cell to support DMA/MDMA chain
From: Pierre Yves MORDRET @ 2018-09-28 13:01 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
Add one cell to support DMA/MDMA chaining.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v3:
v2:
* rework content
v1:
* Initial
---
---
Documentation/devicetree/bindings/dma/stm32-dmamux.txt | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
index 1b893b2..5e92b59 100644
--- a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
+++ b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
@@ -4,9 +4,9 @@ Required properties:
- compatible: "st,stm32h7-dmamux"
- reg: Memory map for accessing module
- #dma-cells: Should be set to <3>.
- First parameter is request line number.
- Second is DMA channel configuration
- Third is Fifo threshold
+- First parameter is request line number.
+- Second is DMA channel configuration
+- Third is a 32bit bitfield
For more details about the three cells, please see
stm32-dma.txt documentation binding file
- dma-masters: Phandle pointing to the DMA controllers.
^ permalink raw reply related
* [v3,3/7] dt-bindings: stm32-mdma: Add DMA/MDMA chaining support bindings
From: Pierre Yves MORDRET @ 2018-09-28 13:01 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
This patch adds the description of the 2 properties needed to support M2M
transfer triggered by STM32 DMA when his transfer is complete.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v3:
v2:
* rework content
v1:
* Initial
---
---
Documentation/devicetree/bindings/dma/stm32-mdma.txt | 12 ++++++++----
1 file changed, 8 insertions(+), 4 deletions(-)
diff --git a/Documentation/devicetree/bindings/dma/stm32-mdma.txt b/Documentation/devicetree/bindings/dma/stm32-mdma.txt
index d18772d..27c2812 100644
--- a/Documentation/devicetree/bindings/dma/stm32-mdma.txt
+++ b/Documentation/devicetree/bindings/dma/stm32-mdma.txt
@@ -10,7 +10,7 @@ Required properties:
- interrupts: Should contain the MDMA interrupt.
- clocks: Should contain the input clock of the DMA instance.
- resets: Reference to a reset controller asserting the DMA controller.
-- #dma-cells : Must be <5>. See DMA client paragraph for more details.
+- #dma-cells : Must be <6>. See DMA client paragraph for more details.
Optional properties:
- dma-channels: Number of DMA channels supported by the controller.
@@ -26,7 +26,7 @@ Example:
interrupts = <122>;
clocks = <&timer_clk>;
resets = <&rcc 992>;
- #dma-cells = <5>;
+ #dma-cells = <6>;
dma-channels = <16>;
dma-requests = <32>;
st,ahb-addr-masks = <0x20000000>, <0x00000000>;
@@ -35,8 +35,8 @@ Example:
* DMA client
DMA clients connected to the STM32 MDMA controller must use the format
-described in the dma.txt file, using a five-cell specifier for each channel:
-a phandle to the MDMA controller plus the following five integer cells:
+described in the dma.txt file, using a six-cell specifier for each channel:
+a phandle to the MDMA controller plus the following six integer cells:
1. The request line number
2. The priority level
@@ -76,6 +76,10 @@ a phandle to the MDMA controller plus the following five integer cells:
if no HW ack signal is used by the MDMA client
5. A 32bit mask specifying the value to be written to acknowledge the request
if no HW ack signal is used by the MDMA client
+6. A bitfield value specifying if the MDMA client wants to generate M2M
+ transfer with HW trigger (1) or not (0). This bitfield should be only
+ enabled for M2M transfer triggered by STM32 DMA client. The memory devices
+ involved in this kind of transfer are SRAM and DDR.
Example:
^ permalink raw reply related
* [v3,4/7] dmaengine: stm32-dma: Add DMA/MDMA chaining support
From: Pierre Yves MORDRET @ 2018-09-28 13:01 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
This patch adds support of DMA/MDMA chaining support.
It introduces an intermediate transfer between peripherals and STM32 DMA.
This intermediate transfer is triggered by SW for single M2D transfer and
by STM32 DMA IP for all other modes (sg, cyclic) and direction (D2M).
A generic SRAM allocator is used for this intermediate buffer
Each DMA channel will be able to define its SRAM needs to achieve chaining
feature : (2 ^ order) * PAGE_SIZE.
For cyclic, SRAM buffer is derived from period length (rounded on
PAGE_SIZE).
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v3:
* Solve KBuild warning
v2:
v1:
* Initial
---
---
drivers/dma/stm32-dma.c | 879 ++++++++++++++++++++++++++++++++++++++++++------
1 file changed, 772 insertions(+), 107 deletions(-)
diff --git a/drivers/dma/stm32-dma.c b/drivers/dma/stm32-dma.c
index 379e8d5..85e81c4 100644
--- a/drivers/dma/stm32-dma.c
+++ b/drivers/dma/stm32-dma.c
@@ -15,11 +15,14 @@
#include <linux/dmaengine.h>
#include <linux/dma-mapping.h>
#include <linux/err.h>
+#include <linux/genalloc.h>
#include <linux/init.h>
+#include <linux/iopoll.h>
#include <linux/jiffies.h>
#include <linux/list.h>
#include <linux/module.h>
#include <linux/of.h>
+#include <linux/of_address.h>
#include <linux/of_device.h>
#include <linux/of_dma.h>
#include <linux/platform_device.h>
@@ -118,6 +121,7 @@
#define STM32_DMA_FIFO_THRESHOLD_FULL 0x03
#define STM32_DMA_MAX_DATA_ITEMS 0xffff
+#define STM32_DMA_SRAM_GRANULARITY PAGE_SIZE
/*
* Valid transfer starts from @0 to @0xFFFE leading to unaligned scatter
* gather at boundary. Thus it's safer to round down this value on FIFO
@@ -135,6 +139,12 @@
/* DMA Features */
#define STM32_DMA_THRESHOLD_FTR_MASK GENMASK(1, 0)
#define STM32_DMA_THRESHOLD_FTR_GET(n) ((n) & STM32_DMA_THRESHOLD_FTR_MASK)
+#define STM32_DMA_MDMA_CHAIN_FTR_MASK BIT(2)
+#define STM32_DMA_MDMA_CHAIN_FTR_GET(n) (((n) & STM32_DMA_MDMA_CHAIN_FTR_MASK) \
+ >> 2)
+#define STM32_DMA_MDMA_SRAM_SIZE_MASK GENMASK(4, 3)
+#define STM32_DMA_MDMA_SRAM_SIZE_GET(n) (((n) & STM32_DMA_MDMA_SRAM_SIZE_MASK) \
+ >> 3)
enum stm32_dma_width {
STM32_DMA_BYTE,
@@ -176,15 +186,31 @@ struct stm32_dma_chan_reg {
u32 dma_sfcr;
};
+struct stm32_dma_mdma_desc {
+ struct sg_table sgt;
+ struct dma_async_tx_descriptor *desc;
+};
+
+struct stm32_dma_mdma {
+ struct dma_chan *chan;
+ enum dma_transfer_direction dir;
+ dma_addr_t sram_buf;
+ u32 sram_period;
+ u32 num_sgs;
+};
+
struct stm32_dma_sg_req {
- u32 len;
+ struct scatterlist stm32_sgl_req;
struct stm32_dma_chan_reg chan_reg;
+ struct stm32_dma_mdma_desc m_desc;
};
struct stm32_dma_desc {
struct virt_dma_desc vdesc;
bool cyclic;
u32 num_sgs;
+ dma_addr_t dma_buf;
+ void *dma_buf_cpu;
struct stm32_dma_sg_req sg_req[];
};
@@ -201,6 +227,10 @@ struct stm32_dma_chan {
u32 threshold;
u32 mem_burst;
u32 mem_width;
+ struct stm32_dma_mdma mchan;
+ u32 use_mdma;
+ u32 sram_size;
+ u32 residue_after_drain;
};
struct stm32_dma_device {
@@ -210,6 +240,7 @@ struct stm32_dma_device {
struct reset_control *rst;
bool mem2mem;
struct stm32_dma_chan chan[STM32_DMA_MAX_CHANNELS];
+ struct gen_pool *sram_pool;
};
static struct stm32_dma_device *stm32_dma_get_dev(struct stm32_dma_chan *chan)
@@ -497,11 +528,15 @@ static void stm32_dma_stop(struct stm32_dma_chan *chan)
static int stm32_dma_terminate_all(struct dma_chan *c)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
unsigned long flags;
LIST_HEAD(head);
spin_lock_irqsave(&chan->vchan.lock, flags);
+ if (chan->use_mdma)
+ dmaengine_terminate_async(mchan->chan);
+
if (chan->busy) {
stm32_dma_stop(chan);
chan->desc = NULL;
@@ -514,9 +549,96 @@ static int stm32_dma_terminate_all(struct dma_chan *c)
return 0;
}
+static u32 stm32_dma_get_remaining_bytes(struct stm32_dma_chan *chan)
+{
+ u32 dma_scr, width, ndtr;
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+
+ dma_scr = stm32_dma_read(dmadev, STM32_DMA_SCR(chan->id));
+ width = STM32_DMA_SCR_PSIZE_GET(dma_scr);
+ ndtr = stm32_dma_read(dmadev, STM32_DMA_SNDTR(chan->id));
+
+ return ndtr << width;
+}
+
+static int stm32_dma_mdma_drain(struct stm32_dma_chan *chan)
+{
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+ struct stm32_dma_sg_req *sg_req;
+ struct dma_device *ddev = mchan->chan->device;
+ struct dma_async_tx_descriptor *desc = NULL;
+ enum dma_status status;
+ dma_addr_t src_buf, dst_buf;
+ u32 mdma_residue, mdma_wrote, dma_to_write, len;
+ struct dma_tx_state state;
+ int ret;
+
+ /* DMA/MDMA chain: drain remaining data in SRAM */
+
+ /* Get the residue on MDMA side */
+ status = dmaengine_tx_status(mchan->chan, mchan->chan->cookie, &state);
+ if (status == DMA_COMPLETE)
+ return status;
+
+ mdma_residue = state.residue;
+ sg_req = &chan->desc->sg_req[chan->next_sg - 1];
+ len = sg_dma_len(&sg_req->stm32_sgl_req);
+
+ /*
+ * Total = mdma blocks * sram_period + rest (< sram_period)
+ * so mdma blocks * sram_period = len - mdma residue - rest
+ */
+ mdma_wrote = len - mdma_residue - (len % mchan->sram_period);
+
+ /* Remaining data stuck in SRAM */
+ dma_to_write = mchan->sram_period - stm32_dma_get_remaining_bytes(chan);
+ if (dma_to_write > 0) {
+ /* Stop DMA current operation */
+ stm32_dma_disable_chan(chan);
+
+ /* Terminate current MDMA to initiate a new one */
+ dmaengine_terminate_all(mchan->chan);
+
+ /* Double buffer management */
+ src_buf = mchan->sram_buf +
+ ((mdma_wrote / mchan->sram_period) & 0x1) *
+ mchan->sram_period;
+ dst_buf = sg_dma_address(&sg_req->stm32_sgl_req) + mdma_wrote;
+
+ desc = ddev->device_prep_dma_memcpy(mchan->chan,
+ dst_buf, src_buf,
+ dma_to_write,
+ DMA_PREP_INTERRUPT);
+ if (!desc)
+ return -EINVAL;
+
+ ret = dma_submit_error(dmaengine_submit(desc));
+ if (ret < 0)
+ return ret;
+
+ status = dma_wait_for_async_tx(desc);
+ if (status != DMA_COMPLETE) {
+ dev_err(chan2dev(chan), "flush() dma_wait_for_async_tx error\n");
+ dmaengine_terminate_async(mchan->chan);
+ return -EBUSY;
+ }
+
+ /* We need to store residue for tx_status() */
+ chan->residue_after_drain = len - (mdma_wrote + dma_to_write);
+ }
+
+ return 0;
+}
+
static void stm32_dma_synchronize(struct dma_chan *c)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+
+ if (chan->desc && chan->use_mdma && mchan->dir == DMA_DEV_TO_MEM)
+ if (stm32_dma_mdma_drain(chan))
+ dev_err(chan2dev(chan), "%s: can't drain DMA\n",
+ __func__);
vchan_synchronize(&chan->vchan);
}
@@ -539,62 +661,232 @@ static void stm32_dma_dump_reg(struct stm32_dma_chan *chan)
dev_dbg(chan2dev(chan), "SFCR: 0x%08x\n", sfcr);
}
-static void stm32_dma_configure_next_sg(struct stm32_dma_chan *chan);
-
-static void stm32_dma_start_transfer(struct stm32_dma_chan *chan)
+static int stm32_dma_dummy_memcpy_xfer(struct stm32_dma_chan *chan)
{
struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
- struct virt_dma_desc *vdesc;
+ struct dma_device *ddev = &dmadev->ddev;
+ struct stm32_dma_chan_reg reg;
+ u8 src_buf, dst_buf;
+ dma_addr_t dma_src_buf, dma_dst_buf;
+ u32 ndtr, status;
+ int len, ret;
+
+ ret = 0;
+ src_buf = 0;
+ len = 1;
+
+ dma_src_buf = dma_map_single(ddev->dev, &src_buf, len, DMA_TO_DEVICE);
+ ret = dma_mapping_error(ddev->dev, dma_src_buf);
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "Source buffer map failed\n");
+ return ret;
+ }
+
+ dma_dst_buf = dma_map_single(ddev->dev, &dst_buf, len, DMA_FROM_DEVICE);
+ ret = dma_mapping_error(ddev->dev, dma_dst_buf);
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "Destination buffer map failed\n");
+ dma_unmap_single(ddev->dev, dma_src_buf, len, DMA_TO_DEVICE);
+ return ret;
+ }
+
+ reg.dma_scr = STM32_DMA_SCR_DIR(STM32_DMA_MEM_TO_MEM) |
+ STM32_DMA_SCR_PBURST(STM32_DMA_BURST_SINGLE) |
+ STM32_DMA_SCR_MBURST(STM32_DMA_BURST_SINGLE) |
+ STM32_DMA_SCR_MINC |
+ STM32_DMA_SCR_PINC |
+ STM32_DMA_SCR_TEIE;
+ reg.dma_spar = dma_src_buf;
+ reg.dma_sm0ar = dma_dst_buf;
+ reg.dma_sfcr = STM32_DMA_SFCR_MASK |
+ STM32_DMA_SFCR_FTH(STM32_DMA_FIFO_THRESHOLD_FULL);
+ reg.dma_sm1ar = dma_dst_buf;
+ reg.dma_sndtr = 1;
+
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg.dma_scr);
+ stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg.dma_spar);
+ stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg.dma_sm0ar);
+ stm32_dma_write(dmadev, STM32_DMA_SFCR(chan->id), reg.dma_sfcr);
+ stm32_dma_write(dmadev, STM32_DMA_SM1AR(chan->id), reg.dma_sm1ar);
+ stm32_dma_write(dmadev, STM32_DMA_SNDTR(chan->id), reg.dma_sndtr);
+
+ /* Clear interrupt status if it is there */
+ status = stm32_dma_irq_status(chan);
+ if (status)
+ stm32_dma_irq_clear(chan, status);
+
+ stm32_dma_dump_reg(chan);
+
+ chan->busy = true;
+ /* Start DMA */
+ reg.dma_scr |= STM32_DMA_SCR_EN;
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg.dma_scr);
+
+ ret = readl_relaxed_poll_timeout_atomic(dmadev->base +
+ STM32_DMA_SNDTR(chan->id),
+ ndtr, !ndtr, 10, 1000);
+ if (ret) {
+ dev_err(chan2dev(chan), "%s: timeout!\n", __func__);
+ ret = -EBUSY;
+ }
+
+ chan->busy = false;
+
+ ret = stm32_dma_disable_chan(chan);
+ status = stm32_dma_irq_status(chan);
+ if (status)
+ stm32_dma_irq_clear(chan, status);
+
+ dma_unmap_single(ddev->dev, dma_src_buf, len, DMA_TO_DEVICE);
+ dma_unmap_single(ddev->dev, dma_dst_buf, len, DMA_FROM_DEVICE);
+
+ return ret;
+}
+
+static int stm32_dma_mdma_flush_remaining(struct stm32_dma_chan *chan)
+{
+ struct stm32_dma_mdma *mchan = &chan->mchan;
struct stm32_dma_sg_req *sg_req;
- struct stm32_dma_chan_reg *reg;
- u32 status;
+ struct dma_device *ddev = mchan->chan->device;
+ struct dma_async_tx_descriptor *desc = NULL;
+ enum dma_status status;
+ dma_addr_t src_buf, dst_buf;
+ u32 residue, remain, len;
int ret;
- ret = stm32_dma_disable_chan(chan);
- if (ret < 0)
- return;
+ sg_req = &chan->desc->sg_req[chan->next_sg - 1];
- if (!chan->desc) {
- vdesc = vchan_next_desc(&chan->vchan);
- if (!vdesc)
- return;
+ residue = stm32_dma_get_remaining_bytes(chan);
+ len = sg_dma_len(&sg_req->stm32_sgl_req);
+ remain = len % mchan->sram_period;
- chan->desc = to_stm32_dma_desc(vdesc);
- chan->next_sg = 0;
+ if (residue > 0 && len > mchan->sram_period &&
+ ((len % mchan->sram_period) != 0)) {
+ unsigned long dma_sync_wait_timeout =
+ jiffies + msecs_to_jiffies(5000);
+
+ while (residue > 0 &&
+ residue > (mchan->sram_period - remain)) {
+ if (time_after_eq(jiffies, dma_sync_wait_timeout)) {
+ dev_err(chan2dev(chan),
+ "%s timeout waiting for last bytes\n",
+ __func__);
+ break;
+ }
+ cpu_relax();
+ residue = stm32_dma_get_remaining_bytes(chan);
+ }
+ stm32_dma_disable_chan(chan);
+
+ src_buf = mchan->sram_buf + ((len / mchan->sram_period) & 0x1)
+ * mchan->sram_period;
+ dst_buf = sg_dma_address(&sg_req->stm32_sgl_req) + len -
+ (len % mchan->sram_period);
+
+ desc = ddev->device_prep_dma_memcpy(mchan->chan,
+ dst_buf, src_buf,
+ len % mchan->sram_period,
+ DMA_PREP_INTERRUPT);
+
+ if (!desc)
+ return -EINVAL;
+
+ ret = dma_submit_error(dmaengine_submit(desc));
+ if (ret < 0)
+ return ret;
+
+ status = dma_wait_for_async_tx(desc);
+ if (status != DMA_COMPLETE) {
+ dmaengine_terminate_async(mchan->chan);
+ return -EBUSY;
+ }
}
- if (chan->next_sg == chan->desc->num_sgs)
- chan->next_sg = 0;
+ return 0;
+}
- sg_req = &chan->desc->sg_req[chan->next_sg];
- reg = &sg_req->chan_reg;
+static void stm32_dma_start_transfer(struct stm32_dma_chan *chan);
- stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
- stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg->dma_spar);
- stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg->dma_sm0ar);
- stm32_dma_write(dmadev, STM32_DMA_SFCR(chan->id), reg->dma_sfcr);
- stm32_dma_write(dmadev, STM32_DMA_SM1AR(chan->id), reg->dma_sm1ar);
- stm32_dma_write(dmadev, STM32_DMA_SNDTR(chan->id), reg->dma_sndtr);
+static void stm32_mdma_chan_complete(void *param,
+ const struct dmaengine_result *result)
+{
+ struct stm32_dma_chan *chan = param;
- chan->next_sg++;
+ chan->busy = false;
+ if (result->result == DMA_TRANS_NOERROR) {
+ if (stm32_dma_mdma_flush_remaining(chan)) {
+ dev_err(chan2dev(chan), "Can't flush DMA\n");
+ return;
+ }
- /* Clear interrupt status if it is there */
- status = stm32_dma_irq_status(chan);
- if (status)
- stm32_dma_irq_clear(chan, status);
+ if (chan->next_sg == chan->desc->num_sgs) {
+ list_del(&chan->desc->vdesc.node);
+ vchan_cookie_complete(&chan->desc->vdesc);
+ chan->desc = NULL;
+ }
+ stm32_dma_start_transfer(chan);
+ } else {
+ dev_err(chan2dev(chan), "MDMA transfer error: %d\n",
+ result->result);
+ }
+}
- if (chan->desc->cyclic)
- stm32_dma_configure_next_sg(chan);
+static int stm32_dma_mdma_start(struct stm32_dma_chan *chan,
+ struct stm32_dma_sg_req *sg_req)
+{
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+ struct stm32_dma_mdma_desc *m_desc = &sg_req->m_desc;
+ struct dma_slave_config config;
+ int ret;
- stm32_dma_dump_reg(chan);
+ /* Configure MDMA channel */
+ memset(&config, 0, sizeof(config));
+ if (mchan->dir == DMA_MEM_TO_DEV)
+ config.dst_addr = mchan->sram_buf;
+ else
+ config.src_addr = mchan->sram_buf;
- /* Start DMA */
- reg->dma_scr |= STM32_DMA_SCR_EN;
- stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
+ ret = dmaengine_slave_config(mchan->chan, &config);
+ if (ret < 0)
+ goto error;
+
+ /* Prepare MDMA descriptor */
+ m_desc->desc = dmaengine_prep_slave_sg(mchan->chan, m_desc->sgt.sgl,
+ m_desc->sgt.nents, mchan->dir,
+ DMA_PREP_INTERRUPT);
+ if (!m_desc->desc) {
+ ret = -EINVAL;
+ goto error;
+ }
- chan->busy = true;
+ if (mchan->dir != DMA_MEM_TO_DEV) {
+ m_desc->desc->callback_result = stm32_mdma_chan_complete;
+ m_desc->desc->callback_param = chan;
+ }
- dev_dbg(chan2dev(chan), "vchan %pK: started\n", &chan->vchan);
+ ret = dma_submit_error(dmaengine_submit(m_desc->desc));
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "MDMA submit failed\n");
+ goto error;
+ }
+
+ dma_async_issue_pending(mchan->chan);
+
+ /*
+ * In case of M2D transfer, we have to generate dummy DMA transfer to
+ * copy 1st sg data into SRAM
+ */
+ if (mchan->dir == DMA_MEM_TO_DEV) {
+ ret = stm32_dma_dummy_memcpy_xfer(chan);
+ if (ret < 0) {
+ dmaengine_terminate_async(mchan->chan);
+ goto error;
+ }
+ }
+
+ return 0;
+error:
+ return ret;
}
static void stm32_dma_configure_next_sg(struct stm32_dma_chan *chan)
@@ -626,23 +918,132 @@ static void stm32_dma_configure_next_sg(struct stm32_dma_chan *chan)
}
}
-static void stm32_dma_handle_chan_done(struct stm32_dma_chan *chan)
+static void stm32_dma_start_transfer(struct stm32_dma_chan *chan)
{
- if (chan->desc) {
- if (chan->desc->cyclic) {
- vchan_cyclic_callback(&chan->desc->vdesc);
- chan->next_sg++;
- stm32_dma_configure_next_sg(chan);
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct virt_dma_desc *vdesc;
+ struct stm32_dma_sg_req *sg_req;
+ struct stm32_dma_chan_reg *reg;
+ u32 status;
+ int ret;
+
+ ret = stm32_dma_disable_chan(chan);
+ if (ret < 0)
+ return;
+
+ if (!chan->desc) {
+ vdesc = vchan_next_desc(&chan->vchan);
+ if (!vdesc)
+ return;
+
+ chan->desc = to_stm32_dma_desc(vdesc);
+ chan->next_sg = 0;
+ } else {
+ vdesc = &chan->desc->vdesc;
+ }
+
+ if (chan->next_sg == chan->desc->num_sgs)
+ chan->next_sg = 0;
+
+ sg_req = &chan->desc->sg_req[chan->next_sg];
+ reg = &sg_req->chan_reg;
+
+ /* Clear interrupt status if it is there */
+ status = stm32_dma_irq_status(chan);
+ if (status)
+ stm32_dma_irq_clear(chan, status);
+
+ if (chan->use_mdma) {
+ if (chan->next_sg == 0) {
+ struct stm32_dma_mdma_desc *m_desc;
+
+ m_desc = &sg_req->m_desc;
+ if (chan->desc->cyclic) {
+ /*
+ * If one callback is set, it will be called by
+ * MDMA driver.
+ */
+ if (vdesc->tx.callback) {
+ m_desc->desc->callback =
+ vdesc->tx.callback;
+ m_desc->desc->callback_param =
+ vdesc->tx.callback_param;
+ vdesc->tx.callback = NULL;
+ vdesc->tx.callback_param = NULL;
+ }
+ }
+ }
+
+ if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ ret = stm32_dma_dummy_memcpy_xfer(chan);
+ if (ret < 0) {
+ dmaengine_terminate_async(chan->mchan.chan);
+ chan->desc = NULL;
+ return;
+ }
} else {
- chan->busy = false;
- if (chan->next_sg == chan->desc->num_sgs) {
- list_del(&chan->desc->vdesc.node);
- vchan_cookie_complete(&chan->desc->vdesc);
+ reg->dma_scr &= ~STM32_DMA_SCR_TCIE;
+ }
+
+ if (!chan->desc->cyclic) {
+ /* MDMA already started */
+ if (chan->mchan.dir != DMA_MEM_TO_DEV &&
+ sg_dma_len(&sg_req->stm32_sgl_req) >
+ chan->mchan.sram_period)
+ reg->dma_scr |= STM32_DMA_SCR_DBM;
+ ret = stm32_dma_mdma_start(chan, sg_req);
+ if (ret < 0) {
chan->desc = NULL;
+ return;
}
- stm32_dma_start_transfer(chan);
}
}
+
+ chan->next_sg++;
+
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
+ stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg->dma_spar);
+ stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg->dma_sm0ar);
+ stm32_dma_write(dmadev, STM32_DMA_SFCR(chan->id), reg->dma_sfcr);
+ stm32_dma_write(dmadev, STM32_DMA_SM1AR(chan->id), reg->dma_sm1ar);
+ stm32_dma_write(dmadev, STM32_DMA_SNDTR(chan->id), reg->dma_sndtr);
+
+ if (chan->desc->cyclic)
+ stm32_dma_configure_next_sg(chan);
+
+ stm32_dma_dump_reg(chan);
+
+ /* Start DMA */
+ chan->busy = true;
+ reg->dma_scr |= STM32_DMA_SCR_EN;
+ stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
+
+ dev_dbg(chan2dev(chan), "vchan %pK: started\n", &chan->vchan);
+}
+
+static void stm32_dma_handle_chan_done(struct stm32_dma_chan *chan)
+{
+ if (!chan->desc)
+ return;
+
+ if (chan->desc->cyclic) {
+ vchan_cyclic_callback(&chan->desc->vdesc);
+ if (chan->use_mdma)
+ return;
+ chan->next_sg++;
+ stm32_dma_configure_next_sg(chan);
+ } else {
+ chan->busy = false;
+ if (chan->use_mdma && chan->mchan.dir != DMA_MEM_TO_DEV)
+ return;
+ if (chan->next_sg == chan->desc->num_sgs) {
+ list_del(&chan->desc->vdesc.node);
+ vchan_cookie_complete(&chan->desc->vdesc);
+ chan->desc = NULL;
+ }
+
+ stm32_dma_start_transfer(chan);
+ }
}
static irqreturn_t stm32_dma_chan_irq(int irq, void *devid)
@@ -695,7 +1096,6 @@ static void stm32_dma_issue_pending(struct dma_chan *c)
if (vchan_issue_pending(&chan->vchan) && !chan->desc && !chan->busy) {
dev_dbg(chan2dev(chan), "vchan %pK: issued\n", &chan->vchan);
stm32_dma_start_transfer(chan);
-
}
spin_unlock_irqrestore(&chan->vchan.lock, flags);
}
@@ -836,16 +1236,128 @@ static void stm32_dma_clear_reg(struct stm32_dma_chan_reg *regs)
memset(regs, 0, sizeof(struct stm32_dma_chan_reg));
}
+static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
+ struct scatterlist *sgl, u32 sg_len,
+ struct stm32_dma_desc *desc)
+{
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct scatterlist *sg, *m_sg;
+ dma_addr_t dma_buf;
+ u32 len, num_sgs, sram_period;
+ int i, j, ret;
+
+ desc->dma_buf_cpu = gen_pool_dma_alloc(dmadev->sram_pool,
+ chan->sram_size,
+ &desc->dma_buf);
+ if (!desc->dma_buf_cpu)
+ return -ENOMEM;
+
+ sram_period = chan->sram_size / 2;
+
+ for_each_sg(sgl, sg, sg_len, i) {
+ struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[i].m_desc;
+
+ len = sg_dma_len(sg);
+ desc->sg_req[i].stm32_sgl_req = *sg;
+ num_sgs = 1;
+
+ if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ if (len > chan->sram_size) {
+ dev_err(chan2dev(chan),
+ "max buf size = %d bytes\n",
+ chan->sram_size);
+ goto free_alloc;
+ }
+ } else {
+ /*
+ * Build new sg for MDMA transfer
+ * Scatter DMA Req into several SDRAM transfer
+ */
+ if (len > sram_period)
+ num_sgs = len / sram_period;
+ }
+
+ ret = sg_alloc_table(&m_desc->sgt, num_sgs, GFP_ATOMIC);
+ if (ret) {
+ dev_err(chan2dev(chan), "MDMA sg table alloc failed\n");
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ dma_buf = sg_dma_address(sg);
+ for_each_sg(m_desc->sgt.sgl, m_sg, num_sgs, j) {
+ size_t bytes = min_t(size_t, len, sram_period);
+
+ sg_dma_address(m_sg) = dma_buf;
+ sg_dma_len(m_sg) = bytes;
+ dma_buf += bytes;
+ len -= bytes;
+ }
+ }
+
+ chan->mchan.sram_buf = desc->dma_buf;
+ chan->mchan.sram_period = sram_period;
+ chan->mchan.num_sgs = num_sgs;
+
+ return 0;
+
+err:
+ for (j = 0; j < i; j++)
+ sg_free_table(&desc->sg_req[j].m_desc.sgt);
+free_alloc:
+ gen_pool_free(dmadev->sram_pool, (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ return ret;
+}
+
+static int stm32_dma_setup_sg_requests(struct stm32_dma_chan *chan,
+ struct scatterlist *sgl,
+ unsigned int sg_len,
+ enum dma_transfer_direction direction,
+ struct stm32_dma_desc *desc)
+{
+ struct scatterlist *sg;
+ u32 nb_data_items;
+ int i, ret;
+ enum dma_slave_buswidth buswidth;
+
+ for_each_sg(sgl, sg, sg_len, i) {
+ ret = stm32_dma_set_xfer_param(chan, direction, &buswidth,
+ sg_dma_len(sg));
+ if (ret < 0)
+ return ret;
+
+ nb_data_items = sg_dma_len(sg) / buswidth;
+ if (nb_data_items > STM32_DMA_ALIGNED_MAX_DATA_ITEMS) {
+ dev_err(chan2dev(chan), "nb items not supported\n");
+ return -EINVAL;
+ }
+
+ stm32_dma_clear_reg(&desc->sg_req[i].chan_reg);
+ desc->sg_req[i].chan_reg.dma_scr = chan->chan_reg.dma_scr;
+ desc->sg_req[i].chan_reg.dma_sfcr = chan->chan_reg.dma_sfcr;
+ desc->sg_req[i].chan_reg.dma_spar = chan->chan_reg.dma_spar;
+ desc->sg_req[i].chan_reg.dma_sm0ar = sg_dma_address(sg);
+ desc->sg_req[i].chan_reg.dma_sm1ar = sg_dma_address(sg);
+ if (chan->use_mdma)
+ desc->sg_req[i].chan_reg.dma_sm1ar +=
+ chan->mchan.sram_period;
+ desc->sg_req[i].chan_reg.dma_sndtr = nb_data_items;
+ }
+
+ desc->num_sgs = sg_len;
+
+ return 0;
+}
+
static struct dma_async_tx_descriptor *stm32_dma_prep_slave_sg(
struct dma_chan *c, struct scatterlist *sgl,
u32 sg_len, enum dma_transfer_direction direction,
unsigned long flags, void *context)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+
struct stm32_dma_desc *desc;
- struct scatterlist *sg;
- enum dma_slave_buswidth buswidth;
- u32 nb_data_items;
int i, ret;
if (!chan->config_init) {
@@ -868,48 +1380,141 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_slave_sg(
else
chan->chan_reg.dma_scr &= ~STM32_DMA_SCR_PFCTRL;
- for_each_sg(sgl, sg, sg_len, i) {
- ret = stm32_dma_set_xfer_param(chan, direction, &buswidth,
- sg_dma_len(sg));
- if (ret < 0)
- goto err;
-
- desc->sg_req[i].len = sg_dma_len(sg);
+ if (chan->use_mdma) {
+ struct sg_table new_sgt;
+ struct scatterlist *s, *_sgl;
- nb_data_items = desc->sg_req[i].len / buswidth;
- if (nb_data_items > STM32_DMA_ALIGNED_MAX_DATA_ITEMS) {
- dev_err(chan2dev(chan), "nb items not supported\n");
- goto err;
+ chan->mchan.dir = direction;
+ ret = stm32_dma_mdma_prep_slave_sg(chan, sgl, sg_len, desc);
+ if (ret < 0)
+ return NULL;
+
+ ret = sg_alloc_table(&new_sgt, sg_len, GFP_ATOMIC);
+ if (ret)
+ dev_err(chan2dev(chan), "DMA sg table alloc failed\n");
+
+ for_each_sg(new_sgt.sgl, s, sg_len, i) {
+ _sgl = sgl;
+ sg_dma_len(s) =
+ min(sg_dma_len(_sgl), chan->mchan.sram_period);
+ s->dma_address = chan->mchan.sram_buf;
+ _sgl = sg_next(_sgl);
}
- stm32_dma_clear_reg(&desc->sg_req[i].chan_reg);
- desc->sg_req[i].chan_reg.dma_scr = chan->chan_reg.dma_scr;
- desc->sg_req[i].chan_reg.dma_sfcr = chan->chan_reg.dma_sfcr;
- desc->sg_req[i].chan_reg.dma_spar = chan->chan_reg.dma_spar;
- desc->sg_req[i].chan_reg.dma_sm0ar = sg_dma_address(sg);
- desc->sg_req[i].chan_reg.dma_sm1ar = sg_dma_address(sg);
- desc->sg_req[i].chan_reg.dma_sndtr = nb_data_items;
+ ret = stm32_dma_setup_sg_requests(chan, new_sgt.sgl, sg_len,
+ direction, desc);
+ sg_free_table(&new_sgt);
+ if (ret < 0)
+ goto err;
+ } else {
+ /* Prepare a normal DMA transfer */
+ ret = stm32_dma_setup_sg_requests(chan, sgl, sg_len, direction,
+ desc);
+ if (ret < 0)
+ goto err;
}
- desc->num_sgs = sg_len;
desc->cyclic = false;
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
-
err:
+ if (chan->use_mdma) {
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+
+ for (i = 0; i < sg_len; i++)
+ sg_free_table(&desc->sg_req[i].m_desc.sgt);
+
+ gen_pool_free(dmadev->sram_pool,
+ (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ }
kfree(desc);
+
return NULL;
}
+static int stm32_dma_mdma_prep_dma_cyclic(struct stm32_dma_chan *chan,
+ dma_addr_t buf_addr, size_t buf_len,
+ size_t period_len,
+ struct stm32_dma_desc *desc)
+{
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
+ struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[0].m_desc;
+ struct dma_slave_config config;
+ dma_addr_t mem;
+ int ret;
+
+ chan->sram_size = ALIGN(period_len, STM32_DMA_SRAM_GRANULARITY);
+ desc->dma_buf_cpu = gen_pool_dma_alloc(dmadev->sram_pool,
+ 2 * chan->sram_size,
+ &desc->dma_buf);
+ if (!desc->dma_buf_cpu)
+ return -ENOMEM;
+
+ memset(&config, 0, sizeof(config));
+ mem = buf_addr;
+
+ /* Configure MDMA channel */
+ if (chan->mchan.dir == DMA_MEM_TO_DEV)
+ config.dst_addr = desc->dma_buf;
+ else
+ config.src_addr = desc->dma_buf;
+ ret = dmaengine_slave_config(mchan->chan, &config);
+ if (ret < 0)
+ goto err;
+
+ /* Prepare MDMA descriptor */
+ m_desc->desc = dmaengine_prep_dma_cyclic(mchan->chan, buf_addr, buf_len,
+ period_len, chan->mchan.dir,
+ DMA_PREP_INTERRUPT);
+
+ if (!m_desc->desc) {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ ret = dma_submit_error(dmaengine_submit(m_desc->desc));
+ if (ret < 0) {
+ dev_err(chan2dev(chan), "MDMA submit failed\n");
+ goto err;
+ }
+
+ dma_async_issue_pending(mchan->chan);
+
+ /*
+ * In case of M2D transfer, we have to generate dummy DMA transfer to
+ * copy 1 period of data into SRAM
+ */
+ if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ ret = stm32_dma_dummy_memcpy_xfer(chan);
+ if (ret < 0) {
+ dev_err(chan2dev(chan),
+ "stm32_dma_dummy_memcpy_xfer failed\n");
+ dmaengine_terminate_async(mchan->chan);
+ goto err;
+ }
+ }
+
+ return 0;
+err:
+ gen_pool_free(dmadev->sram_pool,
+ (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ return ret;
+}
+
static struct dma_async_tx_descriptor *stm32_dma_prep_dma_cyclic(
struct dma_chan *c, dma_addr_t buf_addr, size_t buf_len,
size_t period_len, enum dma_transfer_direction direction,
unsigned long flags)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_chan_reg *chan_reg = &chan->chan_reg;
struct stm32_dma_desc *desc;
enum dma_slave_buswidth buswidth;
u32 num_periods, nb_data_items;
+ dma_addr_t dma_buf = 0;
int i, ret;
if (!buf_len || !period_len) {
@@ -957,28 +1562,49 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_dma_cyclic(
/* Clear periph ctrl if client set it */
chan->chan_reg.dma_scr &= ~STM32_DMA_SCR_PFCTRL;
- num_periods = buf_len / period_len;
+ if (chan->use_mdma)
+ num_periods = 1;
+ else
+ num_periods = buf_len / period_len;
desc = stm32_dma_alloc_desc(num_periods);
if (!desc)
return NULL;
- for (i = 0; i < num_periods; i++) {
- desc->sg_req[i].len = period_len;
+ desc->num_sgs = num_periods;
+ desc->cyclic = true;
+ if (chan->use_mdma) {
+ chan->mchan.dir = direction;
+
+ ret = stm32_dma_mdma_prep_dma_cyclic(chan, buf_addr, buf_len,
+ period_len, desc);
+ if (ret < 0)
+ return NULL;
+ dma_buf = desc->dma_buf;
+ } else {
+ dma_buf = buf_addr;
+ }
+
+ for (i = 0; i < num_periods; i++) {
+ sg_dma_len(&desc->sg_req[i].stm32_sgl_req) = period_len;
+ sg_dma_address(&desc->sg_req[i].stm32_sgl_req) = dma_buf;
stm32_dma_clear_reg(&desc->sg_req[i].chan_reg);
- desc->sg_req[i].chan_reg.dma_scr = chan->chan_reg.dma_scr;
- desc->sg_req[i].chan_reg.dma_sfcr = chan->chan_reg.dma_sfcr;
- desc->sg_req[i].chan_reg.dma_spar = chan->chan_reg.dma_spar;
- desc->sg_req[i].chan_reg.dma_sm0ar = buf_addr;
- desc->sg_req[i].chan_reg.dma_sm1ar = buf_addr;
+ desc->sg_req[i].chan_reg.dma_scr = chan_reg->dma_scr;
+ desc->sg_req[i].chan_reg.dma_sfcr = chan_reg->dma_sfcr;
+ desc->sg_req[i].chan_reg.dma_spar = chan_reg->dma_spar;
+ if (chan->use_mdma) {
+ desc->sg_req[i].chan_reg.dma_sm0ar = desc->dma_buf;
+ desc->sg_req[i].chan_reg.dma_sm1ar = desc->dma_buf +
+ chan->sram_size;
+ } else {
+ desc->sg_req[i].chan_reg.dma_sm0ar = dma_buf;
+ desc->sg_req[i].chan_reg.dma_sm1ar = dma_buf;
+ dma_buf += period_len;
+ }
desc->sg_req[i].chan_reg.dma_sndtr = nb_data_items;
- buf_addr += period_len;
}
- desc->num_sgs = num_periods;
- desc->cyclic = true;
-
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
}
@@ -1019,13 +1645,13 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_dma_memcpy(
STM32_DMA_SCR_PINC |
STM32_DMA_SCR_TCIE |
STM32_DMA_SCR_TEIE;
- desc->sg_req[i].chan_reg.dma_sfcr |= STM32_DMA_SFCR_MASK;
+ desc->sg_req[i].chan_reg.dma_sfcr &= ~STM32_DMA_SFCR_MASK;
desc->sg_req[i].chan_reg.dma_sfcr |=
STM32_DMA_SFCR_FTH(threshold);
desc->sg_req[i].chan_reg.dma_spar = src + offset;
desc->sg_req[i].chan_reg.dma_sm0ar = dest + offset;
desc->sg_req[i].chan_reg.dma_sndtr = xfer_count;
- desc->sg_req[i].len = xfer_count;
+ sg_dma_len(&desc->sg_req[i].stm32_sgl_req) = xfer_count;
}
desc->num_sgs = num_sgs;
@@ -1034,18 +1660,6 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_dma_memcpy(
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
}
-static u32 stm32_dma_get_remaining_bytes(struct stm32_dma_chan *chan)
-{
- u32 dma_scr, width, ndtr;
- struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
-
- dma_scr = stm32_dma_read(dmadev, STM32_DMA_SCR(chan->id));
- width = STM32_DMA_SCR_PSIZE_GET(dma_scr);
- ndtr = stm32_dma_read(dmadev, STM32_DMA_SNDTR(chan->id));
-
- return ndtr << width;
-}
-
static size_t stm32_dma_desc_residue(struct stm32_dma_chan *chan,
struct stm32_dma_desc *desc,
u32 next_sg)
@@ -1054,6 +1668,10 @@ static size_t stm32_dma_desc_residue(struct stm32_dma_chan *chan,
u32 residue = 0;
int i;
+ /* Drain case */
+ if (chan->residue_after_drain)
+ return chan->residue_after_drain;
+
/*
* In cyclic mode, for the last period, residue = remaining bytes from
* NDTR
@@ -1069,7 +1687,7 @@ static size_t stm32_dma_desc_residue(struct stm32_dma_chan *chan,
* transferred
*/
for (i = next_sg; i < desc->num_sgs; i++)
- residue += desc->sg_req[i].len;
+ residue += sg_dma_len(&desc->sg_req[i].stm32_sgl_req);
residue += stm32_dma_get_remaining_bytes(chan);
end:
@@ -1089,11 +1707,23 @@ static enum dma_status stm32_dma_tx_status(struct dma_chan *c,
struct dma_tx_state *state)
{
struct stm32_dma_chan *chan = to_stm32_dma_chan(c);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
struct virt_dma_desc *vdesc;
enum dma_status status;
unsigned long flags;
u32 residue = 0;
+ /*
+ * When DMA/MDMA chain is used, we return the status of MDMA in cyclic
+ * mode and for D2M transfer in sg mode in order to return the correct
+ * residue if any
+ */
+ if (chan->desc && chan->use_mdma &&
+ (mchan->dir != DMA_MEM_TO_DEV || chan->desc->cyclic) &&
+ !chan->residue_after_drain)
+ return dmaengine_tx_status(mchan->chan, mchan->chan->cookie,
+ state);
+
status = dma_cookie_status(c, cookie, state);
if (status == DMA_COMPLETE || !state)
return status;
@@ -1155,21 +1785,34 @@ static void stm32_dma_free_chan_resources(struct dma_chan *c)
static void stm32_dma_desc_free(struct virt_dma_desc *vdesc)
{
- kfree(container_of(vdesc, struct stm32_dma_desc, vdesc));
+ struct stm32_dma_desc *desc = to_stm32_dma_desc(vdesc);
+ struct stm32_dma_chan *chan = to_stm32_dma_chan(vdesc->tx.chan);
+ struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ int i;
+
+ if (chan->use_mdma) {
+ for (i = 0; i < desc->num_sgs; i++)
+ sg_free_table(&desc->sg_req[i].m_desc.sgt);
+
+ gen_pool_free(dmadev->sram_pool,
+ (unsigned long)desc->dma_buf_cpu,
+ chan->sram_size);
+ }
+
+ kfree(desc);
}
static void stm32_dma_set_config(struct stm32_dma_chan *chan,
struct stm32_dma_cfg *cfg)
{
stm32_dma_clear_reg(&chan->chan_reg);
-
chan->chan_reg.dma_scr = cfg->stream_config & STM32_DMA_SCR_CFG_MASK;
chan->chan_reg.dma_scr |= STM32_DMA_SCR_REQ(cfg->request_line);
-
- /* Enable Interrupts */
chan->chan_reg.dma_scr |= STM32_DMA_SCR_TEIE | STM32_DMA_SCR_TCIE;
-
chan->threshold = STM32_DMA_THRESHOLD_FTR_GET(cfg->features);
+ chan->use_mdma = STM32_DMA_MDMA_CHAIN_FTR_GET(cfg->features);
+ chan->sram_size = (1 << STM32_DMA_MDMA_SRAM_SIZE_GET(cfg->features)) *
+ STM32_DMA_SRAM_GRANULARITY;
}
static struct dma_chan *stm32_dma_of_xlate(struct of_phandle_args *dma_spec,
@@ -1207,6 +1850,9 @@ static struct dma_chan *stm32_dma_of_xlate(struct of_phandle_args *dma_spec,
stm32_dma_set_config(chan, &cfg);
+ if (!dmadev->sram_pool || !chan->mchan.chan)
+ chan->use_mdma = 0;
+
return c;
}
@@ -1219,10 +1865,12 @@ MODULE_DEVICE_TABLE(of, stm32_dma_of_match);
static int stm32_dma_probe(struct platform_device *pdev)
{
struct stm32_dma_chan *chan;
+ struct stm32_dma_mdma *mchan;
struct stm32_dma_device *dmadev;
struct dma_device *dd;
const struct of_device_id *match;
struct resource *res;
+ char name[4];
int i, ret;
match = of_match_device(stm32_dma_of_match, &pdev->dev);
@@ -1258,6 +1906,13 @@ static int stm32_dma_probe(struct platform_device *pdev)
reset_control_deassert(dmadev->rst);
}
+ dmadev->sram_pool = of_gen_pool_get(pdev->dev.of_node, "sram", 0);
+ if (!dmadev->sram_pool)
+ dev_info(&pdev->dev, "no dma pool: can't use MDMA\n");
+ else
+ dev_dbg(&pdev->dev, "SRAM pool: %zu KiB\n",
+ gen_pool_size(dmadev->sram_pool) / 1024);
+
dma_cap_set(DMA_SLAVE, dd->cap_mask);
dma_cap_set(DMA_PRIVATE, dd->cap_mask);
dma_cap_set(DMA_CYCLIC, dd->cap_mask);
@@ -1293,6 +1948,16 @@ static int stm32_dma_probe(struct platform_device *pdev)
chan->id = i;
chan->vchan.desc_free = stm32_dma_desc_free;
vchan_init(&chan->vchan, dd);
+
+ mchan = &chan->mchan;
+ if (dmadev->sram_pool) {
+ snprintf(name, sizeof(name), "ch%d", chan->id);
+ mchan->chan = dma_request_slave_channel(dd->dev, name);
+ if (!mchan->chan)
+ dev_info(&pdev->dev,
+ "can't request MDMA chan for %s\n",
+ name);
+ }
}
ret = dma_async_device_register(dd);
@@ -1350,4 +2015,4 @@ static int __init stm32_dma_init(void)
{
return platform_driver_probe(&stm32_dma_driver, stm32_dma_probe);
}
-subsys_initcall(stm32_dma_init);
+device_initcall(stm32_dma_init);
^ permalink raw reply related
* [v3,5/7] dmaengine: stm32-mdma: Add DMA/MDMA chaining support
From: Pierre Yves MORDRET @ 2018-09-28 13:01 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
This patch adds support for M2M transfer triggered by STM32 DMA in order to
transfer data from/to SRAM to/from DDR.
Normally, this mode should not be needed as transferring data from/to DDR
is supported by the STM32 DMA.
However, the STM32 DMA don't have the ability to generate burst transfer
on the DDR as it only embeds only a 4-word FIFO although the minimal burst
length on the DDR is 8 words.
Due to this constraint, the STM32 DMA transfers data from/to DDR in a
single way and could lead to pollute the DDR.
To avoid this, we have to use SRAM for all transfers where STM32 DMA is
involved.
So, we need to add an intermediate M2M transfer handled by the MDMA, which
has the ability to generate burst transfer on the DDR, to copy data
from/to SRAM to/from DDR as described below:
For M2D: DDR --> MDMA --> SRAM --> DMA --> IP
For D2M: IP --> DMA --> SRAM --> MDMA --> DDR
This intermediate transfer is triggered by the STM32 DMA when his transfer
complete flag is set. In that way, we are able to build a DMA/MDMA
chaining transfer completely handled by HW.
This patch clearly adds support for M2M transfer triggered by HW.
This mode is not really available in dmaengine framework as normally M2M
transfers are triggered by SW.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v3:
v2:
v1:
* Initial
---
---
drivers/dma/stm32-mdma.c | 131 +++++++++++++++++++++++++++++++++++++++++------
1 file changed, 114 insertions(+), 17 deletions(-)
diff --git a/drivers/dma/stm32-mdma.c b/drivers/dma/stm32-mdma.c
index 06dd172..6b6e63b 100644
--- a/drivers/dma/stm32-mdma.c
+++ b/drivers/dma/stm32-mdma.c
@@ -211,6 +211,8 @@
#define STM32_MDMA_MAX_BURST 128
#define STM32_MDMA_VERY_HIGH_PRIORITY 0x11
+#define STM32_DMA_SRAM_GRANULARITY PAGE_SIZE
+
enum stm32_mdma_trigger_mode {
STM32_MDMA_BUFFER,
STM32_MDMA_BLOCK,
@@ -237,6 +239,7 @@ struct stm32_mdma_chan_config {
u32 transfer_config;
u32 mask_addr;
u32 mask_data;
+ bool m2m_hw;
};
struct stm32_mdma_hwdesc {
@@ -262,6 +265,7 @@ struct stm32_mdma_desc {
u32 ccr;
bool cyclic;
u32 count;
+ enum dma_transfer_direction dir;
struct stm32_mdma_desc_node node[];
};
@@ -577,13 +581,25 @@ static int stm32_mdma_set_xfer_param(struct stm32_mdma_chan *chan,
dst_addr = chan->dma_config.dst_addr;
/* Set device data size */
+ if (chan_config->m2m_hw)
+ dst_addr_width =
+ stm32_mdma_get_max_width(dst_addr, buf_len,
+ STM32_MDMA_MAX_BUF_LEN);
+
dst_bus_width = stm32_mdma_get_width(chan, dst_addr_width);
if (dst_bus_width < 0)
return dst_bus_width;
ctcr &= ~STM32_MDMA_CTCR_DSIZE_MASK;
ctcr |= STM32_MDMA_CTCR_DSIZE(dst_bus_width);
+ if (chan_config->m2m_hw) {
+ ctcr &= ~STM32_MDMA_CTCR_DINCOS_MASK;
+ ctcr |= STM32_MDMA_CTCR_DINCOS(dst_bus_width);
+ }
/* Set device burst value */
+ if (chan_config->m2m_hw)
+ dst_maxburst = STM32_MDMA_MAX_BUF_LEN / dst_addr_width;
+
dst_best_burst = stm32_mdma_get_best_burst(buf_len, tlen,
dst_maxburst,
dst_addr_width);
@@ -626,13 +642,25 @@ static int stm32_mdma_set_xfer_param(struct stm32_mdma_chan *chan,
src_addr = chan->dma_config.src_addr;
/* Set device data size */
+ if (chan_config->m2m_hw)
+ src_addr_width =
+ stm32_mdma_get_max_width(src_addr, buf_len,
+ STM32_MDMA_MAX_BUF_LEN);
+
src_bus_width = stm32_mdma_get_width(chan, src_addr_width);
if (src_bus_width < 0)
return src_bus_width;
ctcr &= ~STM32_MDMA_CTCR_SSIZE_MASK;
ctcr |= STM32_MDMA_CTCR_SSIZE(src_bus_width);
+ if (chan_config->m2m_hw) {
+ ctcr &= ~STM32_MDMA_CTCR_SINCOS_MASK;
+ ctcr |= STM32_MDMA_CTCR_SINCOS(src_bus_width);
+ }
/* Set device burst value */
+ if (chan_config->m2m_hw)
+ src_maxburst = STM32_MDMA_MAX_BUF_LEN / src_addr_width;
+
src_best_burst = stm32_mdma_get_best_burst(buf_len, tlen,
src_maxburst,
src_addr_width);
@@ -740,6 +768,7 @@ static int stm32_mdma_setup_xfer(struct stm32_mdma_chan *chan,
{
struct stm32_mdma_device *dmadev = stm32_mdma_get_dev(chan);
struct dma_slave_config *dma_config = &chan->dma_config;
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct scatterlist *sg;
dma_addr_t src_addr, dst_addr;
u32 ccr, ctcr, ctbr;
@@ -762,6 +791,8 @@ static int stm32_mdma_setup_xfer(struct stm32_mdma_chan *chan,
} else {
src_addr = dma_config->src_addr;
dst_addr = sg_dma_address(sg);
+ if (chan_config->m2m_hw)
+ src_addr += ((i & 1) ? sg_dma_len(sg) : 0);
ret = stm32_mdma_set_xfer_param(chan, direction, &ccr,
&ctcr, &ctbr, dst_addr,
sg_dma_len(sg));
@@ -780,8 +811,6 @@ static int stm32_mdma_setup_xfer(struct stm32_mdma_chan *chan,
/* Enable interrupts */
ccr &= ~STM32_MDMA_CCR_IRQ_MASK;
ccr |= STM32_MDMA_CCR_TEIE | STM32_MDMA_CCR_CTCIE;
- if (sg_len > 1)
- ccr |= STM32_MDMA_CCR_BTIE;
desc->ccr = ccr;
return 0;
@@ -793,7 +822,9 @@ stm32_mdma_prep_slave_sg(struct dma_chan *c, struct scatterlist *sgl,
unsigned long flags, void *context)
{
struct stm32_mdma_chan *chan = to_stm32_mdma_chan(c);
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct stm32_mdma_desc *desc;
+ struct stm32_mdma_hwdesc *hwdesc;
int i, ret;
/*
@@ -815,6 +846,20 @@ stm32_mdma_prep_slave_sg(struct dma_chan *c, struct scatterlist *sgl,
if (ret < 0)
goto xfer_setup_err;
+ /*
+ * In case of M2M HW transfer triggered by STM32 DMA, we do not have to
+ * clear the transfer complete flag by hardware in order to let the
+ * CPU rearm the DMA with the next sg element and update some data in
+ * dmaengine framework
+ */
+ if (chan_config->m2m_hw && direction == DMA_MEM_TO_DEV) {
+ for (i = 0; i < sg_len; i++) {
+ hwdesc = desc->node[i].hwdesc;
+ hwdesc->cmar = 0;
+ hwdesc->cmdr = 0;
+ }
+ }
+
desc->cyclic = false;
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
@@ -836,9 +881,10 @@ stm32_mdma_prep_dma_cyclic(struct dma_chan *c, dma_addr_t buf_addr,
struct stm32_mdma_chan *chan = to_stm32_mdma_chan(c);
struct stm32_mdma_device *dmadev = stm32_mdma_get_dev(chan);
struct dma_slave_config *dma_config = &chan->dma_config;
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct stm32_mdma_desc *desc;
dma_addr_t src_addr, dst_addr;
- u32 ccr, ctcr, ctbr, count;
+ u32 ccr, ctcr, ctbr, count, offset;
int i, ret;
/*
@@ -892,12 +938,29 @@ stm32_mdma_prep_dma_cyclic(struct dma_chan *c, dma_addr_t buf_addr,
desc->ccr = ccr;
/* Configure hwdesc list */
+ offset = ALIGN(period_len, STM32_DMA_SRAM_GRANULARITY);
for (i = 0; i < count; i++) {
if (direction == DMA_MEM_TO_DEV) {
+ /*
+ * When the DMA is configured in double buffer mode,
+ * the MDMA has to use 2 destination buffers to be
+ * compliant with this mode.
+ */
+ if (chan_config->m2m_hw && count > 1 && i % 2)
+ dst_addr = dma_config->dst_addr + offset;
+ else
+ dst_addr = dma_config->dst_addr;
src_addr = buf_addr + i * period_len;
- dst_addr = dma_config->dst_addr;
} else {
- src_addr = dma_config->src_addr;
+ /*
+ * When the DMA is configured in double buffer mode,
+ * the MDMA has to use 2 destination buffers to be
+ * compliant with this mode.
+ */
+ if (chan_config->m2m_hw && count > 1 && i % 2)
+ src_addr = dma_config->src_addr + offset;
+ else
+ src_addr = dma_config->src_addr;
dst_addr = buf_addr + i * period_len;
}
@@ -907,6 +970,7 @@ stm32_mdma_prep_dma_cyclic(struct dma_chan *c, dma_addr_t buf_addr,
}
desc->cyclic = true;
+ desc->dir = direction;
return vchan_tx_prep(&chan->vchan, &desc->vdesc, flags);
@@ -1287,14 +1351,28 @@ static size_t stm32_mdma_desc_residue(struct stm32_mdma_chan *chan,
{
struct stm32_mdma_device *dmadev = stm32_mdma_get_dev(chan);
struct stm32_mdma_hwdesc *hwdesc = desc->node[0].hwdesc;
- u32 cbndtr, residue, modulo, burst_size;
+ u32 residue = 0;
+ u32 modulo, burst_size;
+ dma_addr_t next_clar;
+ u32 cbndtr;
int i;
- residue = 0;
- for (i = curr_hwdesc + 1; i < desc->count; i++) {
+ /*
+ * Get the residue of pending descriptors
+ */
+ /* Get the next hw descriptor to process from current transfer */
+ next_clar = stm32_mdma_read(dmadev, STM32_MDMA_CLAR(chan->id));
+ for (i = desc->count - 1; i >= 0; i--) {
hwdesc = desc->node[i].hwdesc;
+
+ if (hwdesc->clar == next_clar)
+ break;/* Current transfer found, stop cumulating */
+
+ /* Cumulate residue of unprocessed hw descriptors */
residue += STM32_MDMA_CBNDTR_BNDT(hwdesc->cbndtr);
}
+
+ /* Read & cumulate the residue of the current transfer */
cbndtr = stm32_mdma_read(dmadev, STM32_MDMA_CBNDTR(chan->id));
residue += cbndtr & STM32_MDMA_CBNDTR_BNDT_MASK;
@@ -1314,24 +1392,39 @@ static enum dma_status stm32_mdma_tx_status(struct dma_chan *c,
struct dma_tx_state *state)
{
struct stm32_mdma_chan *chan = to_stm32_mdma_chan(c);
+ struct stm32_mdma_chan_config *chan_config = &chan->chan_config;
struct virt_dma_desc *vdesc;
enum dma_status status;
unsigned long flags;
u32 residue = 0;
status = dma_cookie_status(c, cookie, state);
- if ((status == DMA_COMPLETE) || (!state))
+ if (status == DMA_COMPLETE || !state)
return status;
spin_lock_irqsave(&chan->vchan.lock, flags);
vdesc = vchan_find_desc(&chan->vchan, cookie);
- if (chan->desc && cookie == chan->desc->vdesc.tx.cookie)
- residue = stm32_mdma_desc_residue(chan, chan->desc,
- chan->curr_hwdesc);
- else if (vdesc)
+ if (chan->desc && cookie == chan->desc->vdesc.tx.cookie) {
+ /*
+ * In case of M2D transfer triggered by STM32 DMA, the MDMA has
+ * always one period in advance in cyclic mode. So, we have to
+ * add 1 period of data to return the good residue to the
+ * client
+ */
+ if (chan_config->m2m_hw && chan->desc->dir == DMA_MEM_TO_DEV &&
+ chan->curr_hwdesc > 1)
+ residue =
+ stm32_mdma_desc_residue(chan, chan->desc,
+ chan->curr_hwdesc - 1);
+ else
+ residue = stm32_mdma_desc_residue(chan, chan->desc,
+ chan->curr_hwdesc);
+ } else if (vdesc) {
residue = stm32_mdma_desc_residue(chan,
to_stm32_mdma_desc(vdesc), 0);
+ }
+
dma_set_residue(state, residue);
spin_unlock_irqrestore(&chan->vchan.lock, flags);
@@ -1498,7 +1591,7 @@ static struct dma_chan *stm32_mdma_of_xlate(struct of_phandle_args *dma_spec,
struct dma_chan *c;
struct stm32_mdma_chan_config config;
- if (dma_spec->args_count < 5) {
+ if (dma_spec->args_count < 6) {
dev_err(mdma2dev(dmadev), "Bad number of args\n");
return NULL;
}
@@ -1508,6 +1601,7 @@ static struct dma_chan *stm32_mdma_of_xlate(struct of_phandle_args *dma_spec,
config.transfer_config = dma_spec->args[2];
config.mask_addr = dma_spec->args[3];
config.mask_data = dma_spec->args[4];
+ config.m2m_hw = dma_spec->args[5];
if (config.request >= dmadev->nr_requests) {
dev_err(mdma2dev(dmadev), "Bad request line\n");
@@ -1646,19 +1740,20 @@ static int stm32_mdma_probe(struct platform_device *pdev)
dmadev->irq = platform_get_irq(pdev, 0);
if (dmadev->irq < 0) {
dev_err(&pdev->dev, "failed to get IRQ\n");
- return dmadev->irq;
+ ret = dmadev->irq;
+ goto clk_free;
}
ret = devm_request_irq(&pdev->dev, dmadev->irq, stm32_mdma_irq_handler,
0, dev_name(&pdev->dev), dmadev);
if (ret) {
dev_err(&pdev->dev, "failed to request IRQ\n");
- return ret;
+ goto clk_free;
}
ret = dma_async_device_register(dd);
if (ret)
- return ret;
+ goto clk_free;
ret = of_dma_controller_register(of_node, stm32_mdma_of_xlate, dmadev);
if (ret < 0) {
@@ -1675,6 +1770,8 @@ static int stm32_mdma_probe(struct platform_device *pdev)
err_unregister:
dma_async_device_unregister(dd);
+clk_free:
+ clk_disable_unprepare(dmadev->clk);
return ret;
}
^ permalink raw reply related
* [v3,6/7] dmaengine: stm32-dma: enable descriptor_reuse
From: Pierre Yves MORDRET @ 2018-09-28 13:01 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
Enable client to resubmit already processed descriptors
in order to save descriptor creation time.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v3:
v2:
v1:
* Initial
---
---
drivers/dma/stm32-dma.c | 84 +++++++++++++++++++++++++++++++------------------
1 file changed, 54 insertions(+), 30 deletions(-)
diff --git a/drivers/dma/stm32-dma.c b/drivers/dma/stm32-dma.c
index 85e81c4..ba79051 100644
--- a/drivers/dma/stm32-dma.c
+++ b/drivers/dma/stm32-dma.c
@@ -836,34 +836,8 @@ static int stm32_dma_mdma_start(struct stm32_dma_chan *chan,
{
struct stm32_dma_mdma *mchan = &chan->mchan;
struct stm32_dma_mdma_desc *m_desc = &sg_req->m_desc;
- struct dma_slave_config config;
int ret;
- /* Configure MDMA channel */
- memset(&config, 0, sizeof(config));
- if (mchan->dir == DMA_MEM_TO_DEV)
- config.dst_addr = mchan->sram_buf;
- else
- config.src_addr = mchan->sram_buf;
-
- ret = dmaengine_slave_config(mchan->chan, &config);
- if (ret < 0)
- goto error;
-
- /* Prepare MDMA descriptor */
- m_desc->desc = dmaengine_prep_slave_sg(mchan->chan, m_desc->sgt.sgl,
- m_desc->sgt.nents, mchan->dir,
- DMA_PREP_INTERRUPT);
- if (!m_desc->desc) {
- ret = -EINVAL;
- goto error;
- }
-
- if (mchan->dir != DMA_MEM_TO_DEV) {
- m_desc->desc->callback_result = stm32_mdma_chan_complete;
- m_desc->desc->callback_param = chan;
- }
-
ret = dma_submit_error(dmaengine_submit(m_desc->desc));
if (ret < 0) {
dev_err(chan2dev(chan), "MDMA submit failed\n");
@@ -1001,6 +975,7 @@ static void stm32_dma_start_transfer(struct stm32_dma_chan *chan)
chan->next_sg++;
+ reg->dma_scr &= ~STM32_DMA_SCR_EN;
stm32_dma_write(dmadev, STM32_DMA_SCR(chan->id), reg->dma_scr);
stm32_dma_write(dmadev, STM32_DMA_SPAR(chan->id), reg->dma_spar);
stm32_dma_write(dmadev, STM32_DMA_SM0AR(chan->id), reg->dma_sm0ar);
@@ -1238,9 +1213,11 @@ static void stm32_dma_clear_reg(struct stm32_dma_chan_reg *regs)
static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
struct scatterlist *sgl, u32 sg_len,
- struct stm32_dma_desc *desc)
+ struct stm32_dma_desc *desc,
+ unsigned long flags)
{
struct stm32_dma_device *dmadev = stm32_dma_get_dev(chan);
+ struct stm32_dma_mdma *mchan = &chan->mchan;
struct scatterlist *sg, *m_sg;
dma_addr_t dma_buf;
u32 len, num_sgs, sram_period;
@@ -1256,12 +1233,13 @@ static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
for_each_sg(sgl, sg, sg_len, i) {
struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[i].m_desc;
+ struct dma_slave_config config;
len = sg_dma_len(sg);
desc->sg_req[i].stm32_sgl_req = *sg;
num_sgs = 1;
- if (chan->mchan.dir == DMA_MEM_TO_DEV) {
+ if (mchan->dir == DMA_MEM_TO_DEV) {
if (len > chan->sram_size) {
dev_err(chan2dev(chan),
"max buf size = %d bytes\n",
@@ -1293,6 +1271,38 @@ static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
dma_buf += bytes;
len -= bytes;
}
+
+ /* Configure MDMA channel */
+ memset(&config, 0, sizeof(config));
+ if (mchan->dir == DMA_MEM_TO_DEV)
+ config.dst_addr = desc->dma_buf;
+ else
+ config.src_addr = desc->dma_buf;
+
+ ret = dmaengine_slave_config(mchan->chan, &config);
+ if (ret < 0)
+ goto err;
+
+ /* Prepare MDMA descriptor */
+ m_desc->desc = dmaengine_prep_slave_sg(mchan->chan,
+ m_desc->sgt.sgl,
+ m_desc->sgt.nents,
+ mchan->dir,
+ DMA_PREP_INTERRUPT);
+
+ if (!m_desc->desc) {
+ ret = -EINVAL;
+ goto err;
+ }
+
+ if (flags & DMA_CTRL_REUSE)
+ dmaengine_desc_set_reuse(m_desc->desc);
+
+ if (mchan->dir != DMA_MEM_TO_DEV) {
+ m_desc->desc->callback_result =
+ stm32_mdma_chan_complete;
+ m_desc->desc->callback_param = chan;
+ }
}
chan->mchan.sram_buf = desc->dma_buf;
@@ -1302,8 +1312,12 @@ static int stm32_dma_mdma_prep_slave_sg(struct stm32_dma_chan *chan,
return 0;
err:
- for (j = 0; j < i; j++)
+ for (j = 0; j < i; j++) {
+ struct stm32_dma_mdma_desc *m_desc = &desc->sg_req[j].m_desc;
+
+ m_desc->desc = NULL;
sg_free_table(&desc->sg_req[j].m_desc.sgt);
+ }
free_alloc:
gen_pool_free(dmadev->sram_pool, (unsigned long)desc->dma_buf_cpu,
chan->sram_size);
@@ -1385,7 +1399,8 @@ static struct dma_async_tx_descriptor *stm32_dma_prep_slave_sg(
struct scatterlist *s, *_sgl;
chan->mchan.dir = direction;
- ret = stm32_dma_mdma_prep_slave_sg(chan, sgl, sg_len, desc);
+ ret = stm32_dma_mdma_prep_slave_sg(chan, sgl, sg_len, desc,
+ flags);
if (ret < 0)
return NULL;
@@ -1791,6 +1806,14 @@ static void stm32_dma_desc_free(struct virt_dma_desc *vdesc)
int i;
if (chan->use_mdma) {
+ struct stm32_dma_mdma_desc *m_desc;
+
+ for (i = 0; i < desc->num_sgs; i++) {
+ m_desc = &desc->sg_req[i].m_desc;
+ dmaengine_desc_free(m_desc->desc);
+ m_desc->desc = NULL;
+ }
+
for (i = 0; i < desc->num_sgs; i++)
sg_free_table(&desc->sg_req[i].m_desc.sgt);
@@ -1934,6 +1957,7 @@ static int stm32_dma_probe(struct platform_device *pdev)
dd->directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
dd->residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
dd->max_burst = STM32_DMA_MAX_BURST;
+ dd->descriptor_reuse = true;
dd->dev = &pdev->dev;
INIT_LIST_HEAD(&dd->channels);
^ permalink raw reply related
* [v3,7/7] dmaengine: stm32-mdma: enable descriptor_reuse
From: Pierre Yves MORDRET @ 2018-09-28 13:01 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
enable reuse to spare descriptors creation on critical UC.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v3:
v2:
v1:
* Initial
---
---
drivers/dma/stm32-mdma.c | 2 ++
1 file changed, 2 insertions(+)
diff --git a/drivers/dma/stm32-mdma.c b/drivers/dma/stm32-mdma.c
index 6b6e63b..80a17dd 100644
--- a/drivers/dma/stm32-mdma.c
+++ b/drivers/dma/stm32-mdma.c
@@ -1715,6 +1715,8 @@ static int stm32_mdma_probe(struct platform_device *pdev)
dd->device_resume = stm32_mdma_resume;
dd->device_terminate_all = stm32_mdma_terminate_all;
dd->device_synchronize = stm32_mdma_synchronize;
+ dd->descriptor_reuse = true;
+
dd->src_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) |
BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) |
BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) |
^ permalink raw reply related
* [1/7] dmaengine: sprd: Remove direction usage from struct dma_slave_config
From: Baolin Wang @ 2018-09-29 5:48 UTC (permalink / raw)
To: dan.j.williams, vkoul, eric.long
Cc: broonie, baolin.wang, dmaengine, linux-kernel
The direction field of struct dma_slave_config was marked deprecated,
thus remove the usage.
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/dma/sprd-dma.c | 3 ---
1 file changed, 3 deletions(-)
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index 38d4e4f..c226dc93 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -847,9 +847,6 @@ static int sprd_dma_slave_config(struct dma_chan *chan,
struct sprd_dma_chn *schan = to_sprd_dma_chan(chan);
struct dma_slave_config *slave_cfg = &schan->slave_cfg;
- if (!is_slave_direction(config->direction))
- return -EINVAL;
-
memcpy(slave_cfg, config, sizeof(*config));
return 0;
}
^ permalink raw reply related
* [2/7] dmaengine: sprd: Get transfer residue depending on the transfer direction
From: Baolin Wang @ 2018-09-29 5:49 UTC (permalink / raw)
To: dan.j.williams, vkoul, eric.long
Cc: broonie, baolin.wang, dmaengine, linux-kernel
From: Eric Long <eric.long@spreadtrum.com>
Add one field to save the transfer direction for struct sprd_dma_desc,
which is used to get correct transfer residue depending on the transfer
direction.
[Baolin Wang adds one field to present the transfer direction]
Signed-off-by: Eric Long <eric.long@spreadtrum.com>
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/dma/sprd-dma.c | 21 ++++++++++++++++++++-
1 file changed, 20 insertions(+), 1 deletion(-)
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index c226dc93..4f3587b 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -159,6 +159,7 @@ struct sprd_dma_chn_hw {
struct sprd_dma_desc {
struct virt_dma_desc vd;
struct sprd_dma_chn_hw chn_hw;
+ enum dma_transfer_direction dir;
};
/* dma channel description */
@@ -331,6 +332,17 @@ static void sprd_dma_stop_and_disable(struct sprd_dma_chn *schan)
sprd_dma_disable_chn(schan);
}
+static unsigned long sprd_dma_get_src_addr(struct sprd_dma_chn *schan)
+{
+ unsigned long addr, addr_high;
+
+ addr = readl(schan->chn_base + SPRD_DMA_CHN_SRC_ADDR);
+ addr_high = readl(schan->chn_base + SPRD_DMA_CHN_WARP_PTR) &
+ SPRD_DMA_HIGH_ADDR_MASK;
+
+ return addr | (addr_high << SPRD_DMA_HIGH_ADDR_OFFSET);
+}
+
static unsigned long sprd_dma_get_dst_addr(struct sprd_dma_chn *schan)
{
unsigned long addr, addr_high;
@@ -534,7 +546,12 @@ static enum dma_status sprd_dma_tx_status(struct dma_chan *chan,
else
pos = 0;
} else if (schan->cur_desc && schan->cur_desc->vd.tx.cookie == cookie) {
- pos = sprd_dma_get_dst_addr(schan);
+ struct sprd_dma_desc *sdesc = to_sprd_dma_desc(vd);
+
+ if (sdesc->dir == DMA_DEV_TO_MEM)
+ pos = sprd_dma_get_dst_addr(schan);
+ else
+ pos = sprd_dma_get_src_addr(schan);
} else {
pos = 0;
}
@@ -804,6 +821,8 @@ static int sprd_dma_fill_linklist_desc(struct dma_chan *chan,
if (!sdesc)
return NULL;
+ sdesc->dir = dir;
+
for_each_sg(sgl, sg, sglen, i) {
len = sg_dma_len(sg);
^ permalink raw reply related
* [3/7] dmaengine: sprd: Fix the last link-list configuration
From: Baolin Wang @ 2018-09-29 5:49 UTC (permalink / raw)
To: dan.j.williams, vkoul, eric.long
Cc: broonie, baolin.wang, dmaengine, linux-kernel
From: Eric Long <eric.long@spreadtrum.com>
We will pass sglen as 0 configure the last link-list configuration
when filling the descriptor, which will cause the incorrect link-list
configuration. Thus we should check if the sglen is 0 to configure
the correct link-list configuration.
Signed-off-by: Eric Long <eric.long@spreadtrum.com>
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/dma/sprd-dma.c | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index 4f3587b..e6a74dc 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -697,7 +697,8 @@ static int sprd_dma_fill_desc(struct dma_chan *chan,
hw->cfg |= SPRD_DMA_LINKLIST_EN;
/* link-list index */
- temp = (sg_index + 1) % sglen;
+ temp = sglen ? (sg_index + 1) % sglen : 0;
+
/* Next link-list configuration's physical address offset */
temp = temp * sizeof(*hw) + SPRD_DMA_CHN_SRC_ADDR;
/*
^ permalink raw reply related
* [4/7] dmaengine: sprd: Set cur_desc as NULL when free or terminate one dma channel
From: Baolin Wang @ 2018-09-29 5:49 UTC (permalink / raw)
To: dan.j.williams, vkoul, eric.long
Cc: broonie, baolin.wang, dmaengine, linux-kernel
From: Eric Long <eric.long@spreadtrum.com>
It will be failed to start one new transfer if the channel started one
none interrupt transfer before, since we will only set the schan->cur_desc
as NULL depending on the transfer interrupt now. Thus we should set
schan->cur_desc as NULL when free or terminate one dma channel to
avoid this issue.
Signed-off-by: Eric Long <eric.long@spreadtrum.com>
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/dma/sprd-dma.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index e6a74dc..1b39661 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -439,6 +439,7 @@ static void sprd_dma_stop(struct sprd_dma_chn *schan)
sprd_dma_stop_and_disable(schan);
sprd_dma_unset_uid(schan);
sprd_dma_clear_int(schan);
+ schan->cur_desc = NULL;
}
static bool sprd_dma_check_trans_done(struct sprd_dma_desc *sdesc,
^ permalink raw reply related
* [5/7] dmaengine: sprd: Support DMA link-list cyclic callback
From: Baolin Wang @ 2018-09-29 5:49 UTC (permalink / raw)
To: dan.j.williams, vkoul, eric.long
Cc: broonie, baolin.wang, dmaengine, linux-kernel
From: Eric Long <eric.long@spreadtrum.com>
The Spreadtrum DMA link-list mode is always one cyclic transfer,
so we should clear the SPRD_DMA_LLIST_END flag for the link-list
configuration. Moreover add cyclic callback support for the cyclic
transfer.
Signed-off-by: Eric Long <eric.long@spreadtrum.com>
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/dma/sprd-dma.c | 25 ++++++++++++++-----------
1 file changed, 14 insertions(+), 11 deletions(-)
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index 1b39661..cefe42f 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -463,7 +463,7 @@ static irqreturn_t dma_irq_handle(int irq, void *dev_id)
struct sprd_dma_desc *sdesc;
enum sprd_dma_req_mode req_type;
enum sprd_dma_int_type int_type;
- bool trans_done = false;
+ bool trans_done = false, cyclic = false;
u32 i;
while (irq_status) {
@@ -478,13 +478,19 @@ static irqreturn_t dma_irq_handle(int irq, void *dev_id)
sdesc = schan->cur_desc;
- /* Check if the dma request descriptor is done. */
- trans_done = sprd_dma_check_trans_done(sdesc, int_type,
- req_type);
- if (trans_done == true) {
- vchan_cookie_complete(&sdesc->vd);
- schan->cur_desc = NULL;
- sprd_dma_start(schan);
+ /* cyclic mode schedule callback */
+ cyclic = schan->linklist.phy_addr ? true : false;
+ if (cyclic == true) {
+ vchan_cyclic_callback(&sdesc->vd);
+ } else {
+ /* Check if the dma request descriptor is done. */
+ trans_done = sprd_dma_check_trans_done(sdesc, int_type,
+ req_type);
+ if (trans_done == true) {
+ vchan_cookie_complete(&sdesc->vd);
+ schan->cur_desc = NULL;
+ sprd_dma_start(schan);
+ }
}
spin_unlock(&schan->vc.lock);
}
@@ -692,9 +698,6 @@ static int sprd_dma_fill_desc(struct dma_chan *chan,
/* link-list configuration */
if (schan->linklist.phy_addr) {
- if (sg_index == sglen - 1)
- hw->frg_len |= SPRD_DMA_LLIST_END;
-
hw->cfg |= SPRD_DMA_LINKLIST_EN;
/* link-list index */
^ permalink raw reply related
* [6/7] dmaengine: sprd: Support DMA 2-stage transfer mode
From: Baolin Wang @ 2018-09-29 5:49 UTC (permalink / raw)
To: dan.j.williams, vkoul, eric.long
Cc: broonie, baolin.wang, dmaengine, linux-kernel
From: Eric Long <eric.long@spreadtrum.com>
The Spreadtrum DMA controller supports channel 2-stage tansfer mode,
that means we can request 2 dma channels, one for source channel, and
another one for destination channel. Once the source channel's transaction
is done, it will trigger the destination channel's transaction automatically
by hardware signal.
Signed-off-by: Eric Long <eric.long@spreadtrum.com>
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/dma/sprd-dma.c | 98 +++++++++++++++++++++++++++++++++++++++++-
include/linux/dma/sprd-dma.h | 62 ++++++++++++++++++++++++--
2 files changed, 156 insertions(+), 4 deletions(-)
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index cefe42f..50d6569 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -36,6 +36,8 @@
#define SPRD_DMA_GLB_CHN_EN_STS 0x1c
#define SPRD_DMA_GLB_DEBUG_STS 0x20
#define SPRD_DMA_GLB_ARB_SEL_STS 0x24
+#define SPRD_DMA_GLB_2STAGE_GRP1 0x28
+#define SPRD_DMA_GLB_2STAGE_GRP2 0x2c
#define SPRD_DMA_GLB_REQ_UID(uid) (0x4 * ((uid) - 1))
#define SPRD_DMA_GLB_REQ_UID_OFFSET 0x2000
@@ -57,6 +59,18 @@
#define SPRD_DMA_CHN_SRC_BLK_STEP 0x38
#define SPRD_DMA_CHN_DES_BLK_STEP 0x3c
+/* SPRD_DMA_GLB_2STAGE_GRP register definition */
+#define SPRD_DMA_GLB_2STAGE_EN BIT(24)
+#define SPRD_DMA_GLB_CHN_INT_MASK GENMASK(23, 20)
+#define SPRD_DMA_GLB_LIST_DONE_TRG BIT(19)
+#define SPRD_DMA_GLB_TRANS_DONE_TRG BIT(18)
+#define SPRD_DMA_GLB_BLOCK_DONE_TRG BIT(17)
+#define SPRD_DMA_GLB_FRAG_DONE_TRG BIT(16)
+#define SPRD_DMA_GLB_TRG_OFFSET 16
+#define SPRD_DMA_GLB_DEST_CHN_MASK GENMASK(13, 8)
+#define SPRD_DMA_GLB_DEST_CHN_OFFSET 8
+#define SPRD_DMA_GLB_SRC_CHN_MASK GENMASK(5, 0)
+
/* SPRD_DMA_CHN_INTC register definition */
#define SPRD_DMA_INT_MASK GENMASK(4, 0)
#define SPRD_DMA_INT_CLR_OFFSET 24
@@ -118,6 +132,10 @@
#define SPRD_DMA_SRC_TRSF_STEP_OFFSET 0
#define SPRD_DMA_TRSF_STEP_MASK GENMASK(15, 0)
+/* define DMA channel mode & trigger mode mask */
+#define SPRD_DMA_CHN_MODE_MASK GENMASK(7, 0)
+#define SPRD_DMA_TRG_MODE_MASK GENMASK(7, 0)
+
/* define the DMA transfer step type */
#define SPRD_DMA_NONE_STEP 0
#define SPRD_DMA_BYTE_STEP 1
@@ -170,6 +188,8 @@ struct sprd_dma_chn {
struct dma_slave_config slave_cfg;
u32 chn_num;
u32 dev_id;
+ enum sprd_dma_chn_mode chn_mode;
+ enum sprd_dma_trg_mode trg_mode;
struct sprd_dma_desc *cur_desc;
};
@@ -206,6 +226,16 @@ static inline struct sprd_dma_desc *to_sprd_dma_desc(struct virt_dma_desc *vd)
return container_of(vd, struct sprd_dma_desc, vd);
}
+static void sprd_dma_glb_update(struct sprd_dma_dev *sdev, u32 reg,
+ u32 mask, u32 val)
+{
+ u32 orig = readl(sdev->glb_base + reg);
+ u32 tmp;
+
+ tmp = (orig & ~mask) | val;
+ writel(tmp, sdev->glb_base + reg);
+}
+
static void sprd_dma_chn_update(struct sprd_dma_chn *schan, u32 reg,
u32 mask, u32 val)
{
@@ -389,6 +419,49 @@ static enum sprd_dma_req_mode sprd_dma_get_req_type(struct sprd_dma_chn *schan)
return (frag_reg >> SPRD_DMA_REQ_MODE_OFFSET) & SPRD_DMA_REQ_MODE_MASK;
}
+static int sprd_dma_set_2stage_config(struct sprd_dma_chn *schan)
+{
+ struct sprd_dma_dev *sdev = to_sprd_dma_dev(&schan->vc.chan);
+ u32 val, chn = schan->chn_num + 1;
+
+ switch (schan->chn_mode) {
+ case SPRD_DMA_SRC_CHN0:
+ val = chn & SPRD_DMA_GLB_SRC_CHN_MASK;
+ val |= BIT(schan->trg_mode - 1) << SPRD_DMA_GLB_TRG_OFFSET;
+ val |= SPRD_DMA_GLB_2STAGE_EN;
+ sprd_dma_glb_update(sdev, SPRD_DMA_GLB_2STAGE_GRP1, val, val);
+ break;
+
+ case SPRD_DMA_SRC_CHN1:
+ val = chn & SPRD_DMA_GLB_SRC_CHN_MASK;
+ val |= BIT(schan->trg_mode - 1) << SPRD_DMA_GLB_TRG_OFFSET;
+ val |= SPRD_DMA_GLB_2STAGE_EN;
+ sprd_dma_glb_update(sdev, SPRD_DMA_GLB_2STAGE_GRP2, val, val);
+ break;
+
+ case SPRD_DMA_DST_CHN0:
+ val = (chn << SPRD_DMA_GLB_DEST_CHN_OFFSET) &
+ SPRD_DMA_GLB_DEST_CHN_MASK;
+ val |= SPRD_DMA_GLB_2STAGE_EN;
+ sprd_dma_glb_update(sdev, SPRD_DMA_GLB_2STAGE_GRP1, val, val);
+ break;
+
+ case SPRD_DMA_DST_CHN1:
+ val = (chn << SPRD_DMA_GLB_DEST_CHN_OFFSET) &
+ SPRD_DMA_GLB_DEST_CHN_MASK;
+ val |= SPRD_DMA_GLB_2STAGE_EN;
+ sprd_dma_glb_update(sdev, SPRD_DMA_GLB_2STAGE_GRP2, val, val);
+ break;
+
+ default:
+ dev_err(sdev->dma_dev.dev, "invalid channel mode setting %d\n",
+ schan->chn_mode);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
static void sprd_dma_set_chn_config(struct sprd_dma_chn *schan,
struct sprd_dma_desc *sdesc)
{
@@ -423,6 +496,13 @@ static void sprd_dma_start(struct sprd_dma_chn *schan)
schan->cur_desc = to_sprd_dma_desc(vd);
/*
+ * Set 2-stage configuration if the channel starts one 2-stage
+ * transfer.
+ */
+ if (schan->chn_mode && sprd_dma_set_2stage_config(schan))
+ return;
+
+ /*
* Copy the DMA configuration from DMA descriptor to this hardware
* channel.
*/
@@ -617,6 +697,7 @@ static int sprd_dma_fill_desc(struct dma_chan *chan,
{
struct sprd_dma_dev *sdev = to_sprd_dma_dev(chan);
struct sprd_dma_chn *schan = to_sprd_dma_chan(chan);
+ enum sprd_dma_chn_mode chn_mode = schan->chn_mode;
u32 req_mode = (flags >> SPRD_DMA_REQ_SHIFT) & SPRD_DMA_REQ_MODE_MASK;
u32 int_mode = flags & SPRD_DMA_INT_MASK;
int src_datawidth, dst_datawidth, src_step, dst_step;
@@ -628,7 +709,16 @@ static int sprd_dma_fill_desc(struct dma_chan *chan,
dev_err(sdev->dma_dev.dev, "invalid source step\n");
return src_step;
}
- dst_step = SPRD_DMA_NONE_STEP;
+
+ /*
+ * For 2-stage transfer, destination channel step can not be 0,
+ * since destination device is AON IRAM.
+ */
+ if (chn_mode == SPRD_DMA_DST_CHN0 ||
+ chn_mode == SPRD_DMA_DST_CHN1)
+ dst_step = src_step;
+ else
+ dst_step = SPRD_DMA_NONE_STEP;
} else {
dst_step = sprd_dma_get_step(slave_cfg->dst_addr_width);
if (dst_step < 0) {
@@ -855,6 +945,12 @@ static int sprd_dma_fill_linklist_desc(struct dma_chan *chan,
}
}
+ /* Set channel mode and trigger mode for 2-stage transfer */
+ schan->chn_mode =
+ (flags >> SPRD_DMA_CHN_MODE_SHIFT) & SPRD_DMA_CHN_MODE_MASK;
+ schan->trg_mode =
+ (flags >> SPRD_DMA_TRG_MODE_SHIFT) & SPRD_DMA_TRG_MODE_MASK;
+
ret = sprd_dma_fill_desc(chan, &sdesc->chn_hw, 0, 0, src, dst, len,
dir, flags, slave_cfg);
if (ret) {
diff --git a/include/linux/dma/sprd-dma.h b/include/linux/dma/sprd-dma.h
index b42b80e5..ab82df6 100644
--- a/include/linux/dma/sprd-dma.h
+++ b/include/linux/dma/sprd-dma.h
@@ -3,9 +3,65 @@
#ifndef _SPRD_DMA_H_
#define _SPRD_DMA_H_
-#define SPRD_DMA_REQ_SHIFT 16
-#define SPRD_DMA_FLAGS(req_mode, int_type) \
- ((req_mode) << SPRD_DMA_REQ_SHIFT | (int_type))
+#define SPRD_DMA_REQ_SHIFT 8
+#define SPRD_DMA_TRG_MODE_SHIFT 16
+#define SPRD_DMA_CHN_MODE_SHIFT 24
+#define SPRD_DMA_FLAGS(chn_mode, trg_mode, req_mode, int_type) \
+ ((chn_mode) << SPRD_DMA_CHN_MODE_SHIFT | \
+ (trg_mode) << SPRD_DMA_TRG_MODE_SHIFT | \
+ (req_mode) << SPRD_DMA_REQ_SHIFT | (int_type))
+
+/*
+ * The Spreadtrum DMA controller supports channel 2-stage tansfer, that means
+ * we can request 2 dma channels, one for source channel, and another one for
+ * destination channel. Each channel is independent, and has its own
+ * configurations. Once the source channel's transaction is done, it will
+ * trigger the destination channel's transaction automatically by hardware
+ * signal.
+ *
+ * To support 2-stage tansfer, we must configure the channel mode and trigger
+ * mode as below definition.
+ */
+
+/*
+ * enum sprd_dma_chn_mode: define the DMA channel mode for 2-stage transfer
+ * @SPRD_DMA_CHN_MODE_NONE: No channel mode setting which means channel doesn't
+ * support the 2-stage transfer.
+ * @SPRD_DMA_SRC_CHN0: Channel used as source channel 0.
+ * @SPRD_DMA_SRC_CHN1: Channel used as source channel 1.
+ * @SPRD_DMA_DST_CHN0: Channel used as destination channel 0.
+ * @SPRD_DMA_DST_CHN1: Channel used as destination channel 1.
+ *
+ * Now the DMA controller can supports 2 groups 2-stage transfer.
+ */
+enum sprd_dma_chn_mode {
+ SPRD_DMA_CHN_MODE_NONE,
+ SPRD_DMA_SRC_CHN0,
+ SPRD_DMA_SRC_CHN1,
+ SPRD_DMA_DST_CHN0,
+ SPRD_DMA_DST_CHN1,
+};
+
+/*
+ * enum sprd_dma_trg_mode: define the DMA channel trigger mode for 2-stage
+ * transfer
+ * @SPRD_DMA_NO_TRG: No trigger setting.
+ * @SPRD_DMA_FRAG_DONE_TRG: Trigger the transaction of destination channel
+ * automatically once the source channel's fragment request is done.
+ * @SPRD_DMA_BLOCK_DONE_TRG: Trigger the transaction of destination channel
+ * automatically once the source channel's block request is done.
+ * @SPRD_DMA_TRANS_DONE_TRG: Trigger the transaction of destination channel
+ * automatically once the source channel's transfer request is done.
+ * @SPRD_DMA_LIST_DONE_TRG: Trigger the transaction of destination channel
+ * automatically once the source channel's link-list request is done.
+ */
+enum sprd_dma_trg_mode {
+ SPRD_DMA_NO_TRG,
+ SPRD_DMA_FRAG_DONE_TRG,
+ SPRD_DMA_BLOCK_DONE_TRG,
+ SPRD_DMA_TRANS_DONE_TRG,
+ SPRD_DMA_LIST_DONE_TRG,
+};
/*
* enum sprd_dma_req_mode: define the DMA request mode
^ permalink raw reply related
* [7/7] dmaengine: sprd: Add me as one of the module authors
From: Baolin Wang @ 2018-09-29 5:49 UTC (permalink / raw)
To: dan.j.williams, vkoul, eric.long
Cc: broonie, baolin.wang, dmaengine, linux-kernel
From: Eric Long <eric.long@spreadtrum.com>
Add me as one of the module authors.
Signed-off-by: Eric Long <eric.long@spreadtrum.com>
Signed-off-by: Baolin Wang <baolin.wang@linaro.org>
---
drivers/dma/sprd-dma.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/drivers/dma/sprd-dma.c b/drivers/dma/sprd-dma.c
index 50d6569..e2f0167 100644
--- a/drivers/dma/sprd-dma.c
+++ b/drivers/dma/sprd-dma.c
@@ -1226,4 +1226,5 @@ static int __maybe_unused sprd_dma_runtime_resume(struct device *dev)
MODULE_LICENSE("GPL v2");
MODULE_DESCRIPTION("DMA driver for Spreadtrum");
MODULE_AUTHOR("Baolin Wang <baolin.wang@spreadtrum.com>");
+MODULE_AUTHOR("Eric Long <eric.long@spreadtrum.com>");
MODULE_ALIAS("platform:sprd-dma");
^ permalink raw reply related
* [v2,1/3] arm64: dts: actions: s900: Enable Tx DMA for UART5
From: Manivannan Sadhasivam @ 2018-09-29 7:46 UTC (permalink / raw)
To: vkoul, dan.j.williams, afaerber, robh+dt, gregkh, jslaby
Cc: linux-serial, dmaengine, liuwei, 96boards, devicetree,
daniel.thompson, amit.kucheria, linux-arm-kernel, linux-kernel,
hzhang, bdong, manivannanece23, thomas.liau, jeff.chen, pn,
edgar.righi, Manivannan Sadhasivam
Enable Tx DMA for UART5 in Actions Semi S900 SoC.
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
arch/arm64/boot/dts/actions/s900.dtsi | 2 ++
1 file changed, 2 insertions(+)
diff --git a/arch/arm64/boot/dts/actions/s900.dtsi b/arch/arm64/boot/dts/actions/s900.dtsi
index eceba914762c..39af1236f611 100644
--- a/arch/arm64/boot/dts/actions/s900.dtsi
+++ b/arch/arm64/boot/dts/actions/s900.dtsi
@@ -156,6 +156,8 @@
compatible = "actions,s900-uart", "actions,owl-uart";
reg = <0x0 0xe012a000 0x0 0x2000>;
interrupts = <GIC_SPI 34 IRQ_TYPE_LEVEL_HIGH>;
+ dma-names = "tx";
+ dmas = <&dma 26>;
status = "disabled";
};
^ permalink raw reply related
* [v2,2/3] dmaengine: Add Slave and Cyclic mode support for Actions Semi Owl S900 SoC
From: Manivannan Sadhasivam @ 2018-09-29 7:46 UTC (permalink / raw)
To: vkoul, dan.j.williams, afaerber, robh+dt, gregkh, jslaby
Cc: linux-serial, dmaengine, liuwei, 96boards, devicetree,
daniel.thompson, amit.kucheria, linux-arm-kernel, linux-kernel,
hzhang, bdong, manivannanece23, thomas.liau, jeff.chen, pn,
edgar.righi, Manivannan Sadhasivam
Add Slave and Cyclic mode support for Actions Semi Owl S900 SoC. The slave
mode supports bus width of 4 bytes common for all peripherals and 1 byte
specific for UART.
The cyclic mode supports only block mode transfer.
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
drivers/dma/owl-dma.c | 279 ++++++++++++++++++++++++++++++++++++++++--
1 file changed, 272 insertions(+), 7 deletions(-)
diff --git a/drivers/dma/owl-dma.c b/drivers/dma/owl-dma.c
index 7812a6338acd..1d26db4c9229 100644
--- a/drivers/dma/owl-dma.c
+++ b/drivers/dma/owl-dma.c
@@ -21,6 +21,7 @@
#include <linux/mm.h>
#include <linux/module.h>
#include <linux/of_device.h>
+#include <linux/of_dma.h>
#include <linux/slab.h>
#include "virt-dma.h"
@@ -165,6 +166,7 @@ struct owl_dma_lli {
struct owl_dma_txd {
struct virt_dma_desc vd;
struct list_head lli_list;
+ bool cyclic;
};
/**
@@ -191,6 +193,8 @@ struct owl_dma_vchan {
struct virt_dma_chan vc;
struct owl_dma_pchan *pchan;
struct owl_dma_txd *txd;
+ struct dma_slave_config cfg;
+ u8 drq;
};
/**
@@ -336,9 +340,11 @@ static struct owl_dma_lli *owl_dma_alloc_lli(struct owl_dma *od)
static struct owl_dma_lli *owl_dma_add_lli(struct owl_dma_txd *txd,
struct owl_dma_lli *prev,
- struct owl_dma_lli *next)
+ struct owl_dma_lli *next,
+ bool is_cyclic)
{
- list_add_tail(&next->node, &txd->lli_list);
+ if (!is_cyclic)
+ list_add_tail(&next->node, &txd->lli_list);
if (prev) {
prev->hw.next_lli = next->phys;
@@ -351,7 +357,9 @@ static struct owl_dma_lli *owl_dma_add_lli(struct owl_dma_txd *txd,
static inline int owl_dma_cfg_lli(struct owl_dma_vchan *vchan,
struct owl_dma_lli *lli,
dma_addr_t src, dma_addr_t dst,
- u32 len, enum dma_transfer_direction dir)
+ u32 len, enum dma_transfer_direction dir,
+ struct dma_slave_config *sconfig,
+ bool is_cyclic)
{
struct owl_dma_lli_hw *hw = &lli->hw;
u32 mode;
@@ -364,6 +372,32 @@ static inline int owl_dma_cfg_lli(struct owl_dma_vchan *vchan,
OWL_DMA_MODE_DT_DCU | OWL_DMA_MODE_SAM_INC |
OWL_DMA_MODE_DAM_INC;
+ break;
+ case DMA_MEM_TO_DEV:
+ mode |= OWL_DMA_MODE_TS(vchan->drq)
+ | OWL_DMA_MODE_ST_DCU | OWL_DMA_MODE_DT_DEV
+ | OWL_DMA_MODE_SAM_INC | OWL_DMA_MODE_DAM_CONST;
+
+ /*
+ * Hardware only supports 32bit and 8bit buswidth. Since the
+ * default is 32bit, select 8bit only when requested.
+ */
+ if (sconfig->dst_addr_width == DMA_SLAVE_BUSWIDTH_1_BYTE)
+ mode |= OWL_DMA_MODE_NDDBW_8BIT;
+
+ break;
+ case DMA_DEV_TO_MEM:
+ mode |= OWL_DMA_MODE_TS(vchan->drq)
+ | OWL_DMA_MODE_ST_DEV | OWL_DMA_MODE_DT_DCU
+ | OWL_DMA_MODE_SAM_CONST | OWL_DMA_MODE_DAM_INC;
+
+ /*
+ * Hardware only supports 32bit and 8bit buswidth. Since the
+ * default is 32bit, select 8bit only when requested.
+ */
+ if (sconfig->src_addr_width == DMA_SLAVE_BUSWIDTH_1_BYTE)
+ mode |= OWL_DMA_MODE_NDDBW_8BIT;
+
break;
default:
return -EINVAL;
@@ -381,7 +415,10 @@ static inline int owl_dma_cfg_lli(struct owl_dma_vchan *vchan,
OWL_DMA_LLC_SAV_LOAD_NEXT |
OWL_DMA_LLC_DAV_LOAD_NEXT);
- hw->ctrlb = llc_hw_ctrlb(OWL_DMA_INTCTL_SUPER_BLOCK);
+ if (is_cyclic)
+ hw->ctrlb = llc_hw_ctrlb(OWL_DMA_INTCTL_BLOCK);
+ else
+ hw->ctrlb = llc_hw_ctrlb(OWL_DMA_INTCTL_SUPER_BLOCK);
return 0;
}
@@ -443,6 +480,16 @@ static void owl_dma_terminate_pchan(struct owl_dma *od,
spin_unlock_irqrestore(&od->lock, flags);
}
+static void owl_dma_pause_pchan(struct owl_dma_pchan *pchan)
+{
+ pchan_writel(pchan, 1, OWL_DMAX_PAUSE);
+}
+
+static void owl_dma_resume_pchan(struct owl_dma_pchan *pchan)
+{
+ pchan_writel(pchan, 0, OWL_DMAX_PAUSE);
+}
+
static int owl_dma_start_next_txd(struct owl_dma_vchan *vchan)
{
struct owl_dma *od = to_owl_dma(vchan->vc.chan.device);
@@ -464,7 +511,10 @@ static int owl_dma_start_next_txd(struct owl_dma_vchan *vchan)
lli = list_first_entry(&txd->lli_list,
struct owl_dma_lli, node);
- int_ctl = OWL_DMA_INTCTL_SUPER_BLOCK;
+ if (txd->cyclic)
+ int_ctl = OWL_DMA_INTCTL_BLOCK;
+ else
+ int_ctl = OWL_DMA_INTCTL_SUPER_BLOCK;
pchan_writel(pchan, OWL_DMAX_MODE, OWL_DMA_MODE_LME);
pchan_writel(pchan, OWL_DMAX_LINKLIST_CTL,
@@ -627,6 +677,54 @@ static int owl_dma_terminate_all(struct dma_chan *chan)
return 0;
}
+static int owl_dma_config(struct dma_chan *chan,
+ struct dma_slave_config *config)
+{
+ struct owl_dma_vchan *vchan = to_owl_vchan(chan);
+
+ /* Reject definitely invalid configurations */
+ if (config->src_addr_width == DMA_SLAVE_BUSWIDTH_8_BYTES ||
+ config->dst_addr_width == DMA_SLAVE_BUSWIDTH_8_BYTES)
+ return -EINVAL;
+
+ memcpy(&vchan->cfg, config, sizeof(struct dma_slave_config));
+
+ return 0;
+}
+
+static int owl_dma_pause(struct dma_chan *chan)
+{
+ struct owl_dma_vchan *vchan = to_owl_vchan(chan);
+ unsigned long flags;
+
+ spin_lock_irqsave(&vchan->vc.lock, flags);
+
+ owl_dma_pause_pchan(vchan->pchan);
+
+ spin_unlock_irqrestore(&vchan->vc.lock, flags);
+
+ return 0;
+}
+
+static int owl_dma_resume(struct dma_chan *chan)
+{
+ struct owl_dma_vchan *vchan = to_owl_vchan(chan);
+ unsigned long flags;
+
+ if (!vchan->pchan && !vchan->txd)
+ return 0;
+
+ dev_dbg(chan2dev(chan), "vchan %p: resume\n", &vchan->vc);
+
+ spin_lock_irqsave(&vchan->vc.lock, flags);
+
+ owl_dma_resume_pchan(vchan->pchan);
+
+ spin_unlock_irqrestore(&vchan->vc.lock, flags);
+
+ return 0;
+}
+
static u32 owl_dma_getbytes_chan(struct owl_dma_vchan *vchan)
{
struct owl_dma_pchan *pchan;
@@ -754,13 +852,14 @@ static struct dma_async_tx_descriptor
bytes = min_t(size_t, (len - offset), OWL_DMA_FRAME_MAX_LENGTH);
ret = owl_dma_cfg_lli(vchan, lli, src + offset, dst + offset,
- bytes, DMA_MEM_TO_MEM);
+ bytes, DMA_MEM_TO_MEM,
+ &vchan->cfg, txd->cyclic);
if (ret) {
dev_warn(chan2dev(chan), "failed to config lli\n");
goto err_txd_free;
}
- prev = owl_dma_add_lli(txd, prev, lli);
+ prev = owl_dma_add_lli(txd, prev, lli, false);
}
return vchan_tx_prep(&vchan->vc, &txd->vd, flags);
@@ -770,6 +869,133 @@ static struct dma_async_tx_descriptor
return NULL;
}
+static struct dma_async_tx_descriptor
+ *owl_dma_prep_slave_sg(struct dma_chan *chan,
+ struct scatterlist *sgl,
+ unsigned int sg_len,
+ enum dma_transfer_direction dir,
+ unsigned long flags, void *context)
+{
+ struct owl_dma *od = to_owl_dma(chan->device);
+ struct owl_dma_vchan *vchan = to_owl_vchan(chan);
+ struct dma_slave_config *sconfig = &vchan->cfg;
+ struct owl_dma_txd *txd;
+ struct owl_dma_lli *lli, *prev = NULL;
+ struct scatterlist *sg;
+ dma_addr_t addr, src = 0, dst = 0;
+ size_t len;
+ int ret, i;
+
+ txd = kzalloc(sizeof(*txd), GFP_NOWAIT);
+ if (!txd)
+ return NULL;
+
+ INIT_LIST_HEAD(&txd->lli_list);
+
+ for_each_sg(sgl, sg, sg_len, i) {
+ addr = sg_dma_address(sg);
+ len = sg_dma_len(sg);
+
+ if (len > OWL_DMA_FRAME_MAX_LENGTH) {
+ dev_err(od->dma.dev,
+ "frame length exceeds max supported length");
+ goto err_txd_free;
+ }
+
+ lli = owl_dma_alloc_lli(od);
+ if (!lli) {
+ dev_err(chan2dev(chan), "failed to allocate lli");
+ goto err_txd_free;
+ }
+
+ if (dir == DMA_MEM_TO_DEV) {
+ src = addr;
+ dst = sconfig->dst_addr;
+ } else {
+ src = sconfig->src_addr;
+ dst = addr;
+ }
+
+ ret = owl_dma_cfg_lli(vchan, lli, src, dst, len, dir, sconfig,
+ txd->cyclic);
+ if (ret) {
+ dev_warn(chan2dev(chan), "failed to config lli");
+ goto err_txd_free;
+ }
+
+ prev = owl_dma_add_lli(txd, prev, lli, false);
+ }
+
+ return vchan_tx_prep(&vchan->vc, &txd->vd, flags);
+
+err_txd_free:
+ owl_dma_free_txd(od, txd);
+
+ return NULL;
+}
+
+static struct dma_async_tx_descriptor
+ *owl_prep_dma_cyclic(struct dma_chan *chan,
+ dma_addr_t buf_addr, size_t buf_len,
+ size_t period_len,
+ enum dma_transfer_direction dir,
+ unsigned long flags)
+{
+ struct owl_dma *od = to_owl_dma(chan->device);
+ struct owl_dma_vchan *vchan = to_owl_vchan(chan);
+ struct dma_slave_config *sconfig = &vchan->cfg;
+ struct owl_dma_txd *txd;
+ struct owl_dma_lli *lli, *prev = NULL, *first = NULL;
+ dma_addr_t src = 0, dst = 0;
+ unsigned int periods = buf_len / period_len;
+ int ret, i;
+
+ txd = kzalloc(sizeof(*txd), GFP_NOWAIT);
+ if (!txd)
+ return NULL;
+
+ INIT_LIST_HEAD(&txd->lli_list);
+ txd->cyclic = true;
+
+ for (i = 0; i < periods; i++) {
+ lli = owl_dma_alloc_lli(od);
+ if (!lli) {
+ dev_warn(chan2dev(chan), "failed to allocate lli");
+ goto err_txd_free;
+ }
+
+ if (dir == DMA_MEM_TO_DEV) {
+ src = buf_addr + (period_len * i);
+ dst = sconfig->dst_addr;
+ } else if (dir == DMA_DEV_TO_MEM) {
+ src = sconfig->src_addr;
+ dst = buf_addr + (period_len * i);
+ }
+
+ ret = owl_dma_cfg_lli(vchan, lli, src, dst, period_len,
+ dir, sconfig, txd->cyclic);
+ if (ret) {
+ dev_warn(chan2dev(chan), "failed to config lli");
+ goto err_txd_free;
+ }
+
+ if (!first)
+ first = lli;
+
+ prev = owl_dma_add_lli(txd, prev, lli, false);
+ }
+
+ /* close the cyclic list */
+ owl_dma_add_lli(txd, prev, first, true);
+
+ return vchan_tx_prep(&vchan->vc, &txd->vd, flags);
+
+err_txd_free:
+ owl_dma_free_txd(od, txd);
+
+ return NULL;
+}
+
static void owl_dma_free_chan_resources(struct dma_chan *chan)
{
struct owl_dma_vchan *vchan = to_owl_vchan(chan);
@@ -790,6 +1016,27 @@ static inline void owl_dma_free(struct owl_dma *od)
}
}
+static struct dma_chan *owl_dma_of_xlate(struct of_phandle_args *dma_spec,
+ struct of_dma *ofdma)
+{
+ struct owl_dma *od = ofdma->of_dma_data;
+ struct owl_dma_vchan *vchan;
+ struct dma_chan *chan;
+ u8 drq = dma_spec->args[0];
+
+ if (drq > od->nr_vchans)
+ return NULL;
+
+ chan = dma_get_any_slave_channel(&od->dma);
+ if (!chan)
+ return NULL;
+
+ vchan = to_owl_vchan(chan);
+ vchan->drq = drq;
+
+ return chan;
+}
+
static int owl_dma_probe(struct platform_device *pdev)
{
struct device_node *np = pdev->dev.of_node;
@@ -833,12 +1080,19 @@ static int owl_dma_probe(struct platform_device *pdev)
spin_lock_init(&od->lock);
dma_cap_set(DMA_MEMCPY, od->dma.cap_mask);
+ dma_cap_set(DMA_SLAVE, od->dma.cap_mask);
+ dma_cap_set(DMA_CYCLIC, od->dma.cap_mask);
od->dma.dev = &pdev->dev;
od->dma.device_free_chan_resources = owl_dma_free_chan_resources;
od->dma.device_tx_status = owl_dma_tx_status;
od->dma.device_issue_pending = owl_dma_issue_pending;
od->dma.device_prep_dma_memcpy = owl_dma_prep_memcpy;
+ od->dma.device_prep_slave_sg = owl_dma_prep_slave_sg;
+ od->dma.device_prep_dma_cyclic = owl_prep_dma_cyclic;
+ od->dma.device_config = owl_dma_config;
+ od->dma.device_pause = owl_dma_pause;
+ od->dma.device_resume = owl_dma_resume;
od->dma.device_terminate_all = owl_dma_terminate_all;
od->dma.src_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_4_BYTES);
od->dma.dst_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_4_BYTES);
@@ -910,8 +1164,18 @@ static int owl_dma_probe(struct platform_device *pdev)
goto err_pool_free;
}
+ /* Device-tree DMA controller registration */
+ ret = of_dma_controller_register(pdev->dev.of_node,
+ owl_dma_of_xlate, od);
+ if (ret) {
+ dev_err(&pdev->dev, "of_dma_controller_register failed\n");
+ goto err_dma_unregister;
+ }
+
return 0;
+err_dma_unregister:
+ dma_async_device_unregister(&od->dma);
err_pool_free:
clk_disable_unprepare(od->clk);
dma_pool_destroy(od->lli_pool);
@@ -923,6 +1187,7 @@ static int owl_dma_remove(struct platform_device *pdev)
{
struct owl_dma *od = platform_get_drvdata(pdev);
+ of_dma_controller_free(pdev->dev.of_node);
dma_async_device_unregister(&od->dma);
/* Mask all interrupts for this execution environment */
^ permalink raw reply related
* [v2,3/3] tty: serial: Add Tx DMA support for UART in Actions Semi Owl SoCs
From: Manivannan Sadhasivam @ 2018-09-29 7:46 UTC (permalink / raw)
To: vkoul, dan.j.williams, afaerber, robh+dt, gregkh, jslaby
Cc: linux-serial, dmaengine, liuwei, 96boards, devicetree,
daniel.thompson, amit.kucheria, linux-arm-kernel, linux-kernel,
hzhang, bdong, manivannanece23, thomas.liau, jeff.chen, pn,
edgar.righi, Manivannan Sadhasivam
Add Tx DMA support for Actions Semi Owl SoCs. If there is no DMA
property specified in DT, it will fallback to default interrupt mode.
Signed-off-by: Manivannan Sadhasivam <manivannan.sadhasivam@linaro.org>
---
drivers/tty/serial/owl-uart.c | 172 +++++++++++++++++++++++++++++++++-
1 file changed, 171 insertions(+), 1 deletion(-)
diff --git a/drivers/tty/serial/owl-uart.c b/drivers/tty/serial/owl-uart.c
index 29a6dc6a8d23..1b3016db7ae2 100644
--- a/drivers/tty/serial/owl-uart.c
+++ b/drivers/tty/serial/owl-uart.c
@@ -11,6 +11,8 @@
#include <linux/clk.h>
#include <linux/console.h>
#include <linux/delay.h>
+#include <linux/dmaengine.h>
+#include <linux/dma-mapping.h>
#include <linux/io.h>
#include <linux/module.h>
#include <linux/of.h>
@@ -48,6 +50,8 @@
#define OWL_UART_CTL_RXIE BIT(18)
#define OWL_UART_CTL_TXIE BIT(19)
#define OWL_UART_CTL_LBEN BIT(20)
+#define OWL_UART_CTL_DRCR BIT(21)
+#define OWL_UART_CTL_DTCR BIT(22)
#define OWL_UART_STAT_RIP BIT(0)
#define OWL_UART_STAT_TIP BIT(1)
@@ -71,12 +75,21 @@ struct owl_uart_info {
struct owl_uart_port {
struct uart_port port;
struct clk *clk;
+
+ struct dma_chan *tx_ch;
+ dma_addr_t tx_dma_buf;
+ dma_cookie_t dma_tx_cookie;
+ u32 tx_size;
+ bool tx_dma;
+ bool dma_tx_running;
};
#define to_owl_uart_port(prt) container_of(prt, struct owl_uart_port, prt)
static struct owl_uart_port *owl_uart_ports[OWL_UART_PORT_NUM];
+static void owl_uart_dma_start_tx(struct owl_uart_port *owl_port);
+
static inline void owl_uart_write(struct uart_port *port, u32 val, unsigned int off)
{
writel(val, port->membase + off);
@@ -115,6 +128,83 @@ static unsigned int owl_uart_get_mctrl(struct uart_port *port)
return mctrl;
}
+static void owl_uart_dma_tx_callback(void *data)
+{
+ struct owl_uart_port *owl_port = data;
+ struct uart_port *port = &owl_port->port;
+ struct circ_buf *xmit = &port->state->xmit;
+ unsigned long flags;
+ u32 val;
+
+ dma_sync_single_for_cpu(port->dev, owl_port->tx_dma_buf,
+ UART_XMIT_SIZE, DMA_TO_DEVICE);
+
+ spin_lock_irqsave(&port->lock, flags);
+
+ owl_port->dma_tx_running = 0;
+
+ xmit->tail += owl_port->tx_size;
+ xmit->tail &= UART_XMIT_SIZE - 1;
+ port->icount.tx += owl_port->tx_size;
+
+ if (uart_circ_chars_pending(xmit) < WAKEUP_CHARS)
+ uart_write_wakeup(port);
+
+ /* Disable Tx DRQ */
+ val = owl_uart_read(port, OWL_UART_CTL);
+ val &= ~OWL_UART_CTL_TXDE;
+ owl_uart_write(port, val, OWL_UART_CTL);
+
+ /* Clear pending Tx IRQ */
+ val = owl_uart_read(port, OWL_UART_STAT);
+ val |= OWL_UART_STAT_TIP;
+ owl_uart_write(port, val, OWL_UART_STAT);
+
+ if (!uart_circ_empty(xmit) && !uart_tx_stopped(port))
+ owl_uart_dma_start_tx(owl_port);
+
+ spin_unlock_irqrestore(&port->lock, flags);
+}
+
+static void owl_uart_dma_start_tx(struct owl_uart_port *owl_port)
+{
+ struct uart_port *port = &owl_port->port;
+ struct circ_buf *xmit = &port->state->xmit;
+ struct dma_async_tx_descriptor *desc;
+ u32 val;
+
+ if (uart_tx_stopped(port) || uart_circ_empty(xmit) ||
+ owl_port->dma_tx_running)
+ return;
+
+ dma_sync_single_for_device(port->dev, owl_port->tx_dma_buf,
+ UART_XMIT_SIZE, DMA_TO_DEVICE);
+
+ owl_port->tx_size = CIRC_CNT_TO_END(xmit->head, xmit->tail,
+ UART_XMIT_SIZE);
+
+ desc = dmaengine_prep_slave_single(owl_port->tx_ch,
+ owl_port->tx_dma_buf + xmit->tail,
+ owl_port->tx_size, DMA_MEM_TO_DEV,
+ DMA_PREP_INTERRUPT);
+ if (!desc)
+ return;
+
+ desc->callback = owl_uart_dma_tx_callback;
+ desc->callback_param = owl_port;
+
+ /* Enable Tx DRQ */
+ val = owl_uart_read(port, OWL_UART_CTL);
+ val &= ~OWL_UART_CTL_TXIE;
+ val |= OWL_UART_CTL_TXDE | OWL_UART_CTL_DTCR;
+ owl_uart_write(port, val, OWL_UART_CTL);
+
+ /* Start Tx DMA transfer */
+ owl_port->dma_tx_running = true;
+ owl_port->dma_tx_cookie = dmaengine_submit(desc);
+ dma_async_issue_pending(owl_port->tx_ch);
+}
+
static unsigned int owl_uart_tx_empty(struct uart_port *port)
{
unsigned long flags;
@@ -159,6 +249,7 @@ static void owl_uart_stop_tx(struct uart_port *port)
static void owl_uart_start_tx(struct uart_port *port)
{
+ struct owl_uart_port *owl_port = to_owl_uart_port(port);
u32 val;
if (uart_tx_stopped(port)) {
@@ -166,6 +257,11 @@ static void owl_uart_start_tx(struct uart_port *port)
return;
}
+ if (owl_port->tx_dma) {
+ owl_uart_dma_start_tx(owl_port);
+ return;
+ }
+
val = owl_uart_read(port, OWL_UART_STAT);
val |= OWL_UART_STAT_TIP;
owl_uart_write(port, val, OWL_UART_STAT);
@@ -273,13 +369,27 @@ static irqreturn_t owl_uart_irq(int irq, void *dev_id)
return IRQ_HANDLED;
}
+static void owl_dma_channel_free(struct owl_uart_port *owl_port)
+{
+ dmaengine_terminate_all(owl_port->tx_ch);
+ dma_release_channel(owl_port->tx_ch);
+ dma_unmap_single(owl_port->port.dev, owl_port->tx_dma_buf,
+ UART_XMIT_SIZE, DMA_TO_DEVICE);
+ owl_port->dma_tx_running = false;
+ owl_port->tx_ch = NULL;
+}
+
static void owl_uart_shutdown(struct uart_port *port)
{
- u32 val;
+ struct owl_uart_port *owl_port = to_owl_uart_port(port);
unsigned long flags;
+ u32 val;
spin_lock_irqsave(&port->lock, flags);
+ if (owl_port->tx_dma)
+ owl_dma_channel_free(owl_port);
+
val = owl_uart_read(port, OWL_UART_CTL);
val &= ~(OWL_UART_CTL_TXIE | OWL_UART_CTL_RXIE
| OWL_UART_CTL_TXDE | OWL_UART_CTL_RXDE | OWL_UART_CTL_EN);
@@ -290,6 +400,62 @@ static void owl_uart_shutdown(struct uart_port *port)
free_irq(port->irq, port);
}
+static int owl_uart_dma_tx_init(struct uart_port *port)
+{
+ struct owl_uart_port *owl_port = to_owl_uart_port(port);
+ struct device *dev = port->dev;
+ struct dma_slave_config slave_config;
+ int ret;
+
+ owl_port->tx_dma = false;
+
+ /* Request DMA TX channel */
+ owl_port->tx_ch = dma_request_slave_channel(dev, "tx");
+ if (!owl_port->tx_ch) {
+ dev_info(dev, "tx dma alloc failed\n");
+ return -ENODEV;
+ }
+
+ owl_port->tx_dma_buf = dma_map_single(dev,
+ owl_port->port.state->xmit.buf,
+ UART_XMIT_SIZE, DMA_TO_DEVICE);
+ if (dma_mapping_error(dev, owl_port->tx_dma_buf)) {
+ ret = -ENOMEM;
+ goto alloc_err;
+ }
+
+ /* Configure DMA channel */
+ memset(&slave_config, 0, sizeof(slave_config));
+ slave_config.direction = DMA_MEM_TO_DEV;
+ slave_config.dst_addr = port->mapbase + OWL_UART_TXDAT;
+ slave_config.dst_addr_width = DMA_SLAVE_BUSWIDTH_1_BYTE;
+
+ ret = dmaengine_slave_config(owl_port->tx_ch, &slave_config);
+ if (ret < 0) {
+ dev_err(dev, "tx dma channel config failed\n");
+ ret = -ENODEV;
+ goto map_err;
+ }
+
+ /* Use DMA buffer size as the FIFO size */
+ port->fifosize = UART_XMIT_SIZE;
+
+ /* Set DMA flag */
+ owl_port->tx_dma = true;
+ owl_port->dma_tx_running = false;
+
+ return 0;
+
+map_err:
+ dma_unmap_single(dev, owl_port->tx_dma_buf, UART_XMIT_SIZE,
+ DMA_TO_DEVICE);
+alloc_err:
+ dma_release_channel(owl_port->tx_ch);
+ owl_port->tx_ch = NULL;
+
+ return ret;
+}
+
static int owl_uart_startup(struct uart_port *port)
{
u32 val;
@@ -301,6 +467,10 @@ static int owl_uart_startup(struct uart_port *port)
if (ret)
return ret;
+ ret = owl_uart_dma_tx_init(port);
+ if (!ret)
+ dev_info(port->dev, "using DMA for tx\n");
+
spin_lock_irqsave(&port->lock, flags);
val = owl_uart_read(port, OWL_UART_STAT);
^ permalink raw reply related
* [v2,1/4] dmaengine: xilinx_dma: Refactor axidma channel allocation
From: Radhey Shyam Pandey @ 2018-09-29 17:17 UTC (permalink / raw)
To: vkoul, dan.j.williams, michal.simek, appana.durga.rao,
radhey.shyam.pandey
Cc: dmaengine, linux-arm-kernel, linux-kernel
In axidma alloc_chan_resources merge BD and cyclic BD allocation.
Signed-off-by: Radhey Shyam Pandey <radhey.shyam.pandey@xilinx.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
---
Changes for v2:
None
---
drivers/dma/xilinx/xilinx_dma.c | 36 ++++++++++++++++++------------------
1 files changed, 18 insertions(+), 18 deletions(-)
diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
index c124423..06d1632 100644
--- a/drivers/dma/xilinx/xilinx_dma.c
+++ b/drivers/dma/xilinx/xilinx_dma.c
@@ -887,6 +887,24 @@ static int xilinx_dma_alloc_chan_resources(struct dma_chan *dchan)
chan->id);
return -ENOMEM;
}
+ /*
+ * For cyclic DMA mode we need to program the tail Descriptor
+ * register with a value which is not a part of the BD chain
+ * so allocating a desc segment during channel allocation for
+ * programming tail descriptor.
+ */
+ chan->cyclic_seg_v = dma_zalloc_coherent(chan->dev,
+ sizeof(*chan->cyclic_seg_v),
+ &chan->cyclic_seg_p, GFP_KERNEL);
+ if (!chan->cyclic_seg_v) {
+ dev_err(chan->dev,
+ "unable to allocate desc segment for cyclic DMA\n");
+ dma_free_coherent(chan->dev, sizeof(*chan->seg_v) *
+ XILINX_DMA_NUM_DESCS, chan->seg_v,
+ chan->seg_p);
+ return -ENOMEM;
+ }
+ chan->cyclic_seg_v->phys = chan->cyclic_seg_p;
for (i = 0; i < XILINX_DMA_NUM_DESCS; i++) {
chan->seg_v[i].hw.next_desc =
@@ -922,24 +940,6 @@ static int xilinx_dma_alloc_chan_resources(struct dma_chan *dchan)
return -ENOMEM;
}
- if (chan->xdev->dma_config->dmatype == XDMA_TYPE_AXIDMA) {
- /*
- * For cyclic DMA mode we need to program the tail Descriptor
- * register with a value which is not a part of the BD chain
- * so allocating a desc segment during channel allocation for
- * programming tail descriptor.
- */
- chan->cyclic_seg_v = dma_zalloc_coherent(chan->dev,
- sizeof(*chan->cyclic_seg_v),
- &chan->cyclic_seg_p, GFP_KERNEL);
- if (!chan->cyclic_seg_v) {
- dev_err(chan->dev,
- "unable to allocate desc segment for cyclic DMA\n");
- return -ENOMEM;
- }
- chan->cyclic_seg_v->phys = chan->cyclic_seg_p;
- }
-
dma_cookie_init(dchan);
if (chan->xdev->dma_config->dmatype == XDMA_TYPE_AXIDMA) {
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox