* [V3,4/5] dmaengine: dmatest: Add transfer_size parameter
From: Seraj Alijan @ 2018-09-19 19:52 UTC (permalink / raw)
To: vkoul
Cc: dmaengine, dan.j.williams, james.hartley, sifan.naeem, ed.blake,
Seraj Alijan
Existing transfer size "len" is either generated randomly or set to the
size of test_buf_size. In some cases we need to explicitly specify a
transfer size that is different from the buffer size and non aligned to
test the target device's ability to handle unaligned transfers.
This patch adds optional parameter "transfer_size" to allow setting
explicit transfer size for dma transfers.
Signed-off-by: Seraj Alijan <seraj.alijan@sondrel.com>
---
drivers/dma/dmatest.c | 28 ++++++++++++++++++++++------
1 file changed, 22 insertions(+), 6 deletions(-)
diff --git a/drivers/dma/dmatest.c b/drivers/dma/dmatest.c
index 300fe4c..b6e613b 100644
--- a/drivers/dma/dmatest.c
+++ b/drivers/dma/dmatest.c
@@ -83,6 +83,10 @@ static int alignment = -1;
module_param(alignment, int, 0644);
MODULE_PARM_DESC(alignment, "Custom data address alignment taken as 2^(alignment) (default: not used (-1))");
+static unsigned int transfer_size;
+module_param(transfer_size, uint, 0644);
+MODULE_PARM_DESC(transfer_size, "Optional custom transfer size in bytes (default: not used (0))");
+
/**
* struct dmatest_params - test parameters.
* @buf_size: size of the memcpy test buffer
@@ -108,6 +112,7 @@ struct dmatest_params {
bool noverify;
bool norandom;
int alignment;
+ unsigned int transfer_size;
};
/**
@@ -643,15 +648,25 @@ static int dmatest_func(void *data)
break;
}
- if (params->norandom)
+ if (params->transfer_size) {
+ if (params->transfer_size >= params->buf_size) {
+ pr_err("%u-byte transfer size must be lower than %u-buffer size\n",
+ params->transfer_size, params->buf_size);
+ break;
+ }
+ len = params->transfer_size;
+ } else if (params->norandom) {
len = params->buf_size;
- else
+ } else {
len = dmatest_random() % params->buf_size + 1;
+ }
- len = (len >> align) << align;
- if (!len)
- len = 1 << align;
-
+ /* Do not alter transfer size explicitly defined by user */
+ if (!params->transfer_size) {
+ len = (len >> align) << align;
+ if (!len)
+ len = 1 << align;
+ }
total_len += len;
if (params->norandom) {
@@ -1049,6 +1064,7 @@ static void add_threaded_test(struct dmatest_info *info)
params->noverify = noverify;
params->norandom = norandom;
params->alignment = alignment;
+ params->transfer_size = transfer_size;
request_channels(info, DMA_MEMCPY);
request_channels(info, DMA_MEMSET);
^ permalink raw reply related
* [V3,5/5] dmaengine: Documentation: Add documentation for multi chan testing
From: Seraj Alijan @ 2018-09-19 19:52 UTC (permalink / raw)
To: vkoul
Cc: dmaengine, dan.j.williams, james.hartley, sifan.naeem, ed.blake,
Seraj Alijan
Modify documentation to add multi channel testing support.
Signed-off-by: Seraj Alijan <seraj.alijan@sondrel.com>
---
Documentation/driver-api/dmaengine/dmatest.rst | 109 +++++++++++++++++++++++--
1 file changed, 103 insertions(+), 6 deletions(-)
diff --git a/Documentation/driver-api/dmaengine/dmatest.rst b/Documentation/driver-api/dmaengine/dmatest.rst
index 7ce5e71..dd5263a 100644
--- a/Documentation/driver-api/dmaengine/dmatest.rst
+++ b/Documentation/driver-api/dmaengine/dmatest.rst
@@ -26,28 +26,43 @@ Part 2 - When dmatest is built as a module
Example of usage::
- % modprobe dmatest channel=dma0chan0 timeout=2000 iterations=1 run=1
+ % modprobe dmatest timeout=2000 iterations=1 channel=dma0chan0 run=1
...or::
% modprobe dmatest
- % echo dma0chan0 > /sys/module/dmatest/parameters/channel
% echo 2000 > /sys/module/dmatest/parameters/timeout
% echo 1 > /sys/module/dmatest/parameters/iterations
+ % echo dma0chan0 > /sys/module/dmatest/parameters/channel
% echo 1 > /sys/module/dmatest/parameters/run
...or on the kernel command line::
- dmatest.channel=dma0chan0 dmatest.timeout=2000 dmatest.iterations=1 dmatest.run=1
+ dmatest.timeout=2000 dmatest.iterations=1 dmatest.channel=dma0chan0 dmatest.run=1
+
+Example of multi-channel test usage:
+ % modprobe dmatest
+ % echo 2000 > /sys/module/dmatest/parameters/timeout
+ % echo 1 > /sys/module/dmatest/parameters/iterations
+ % echo dma0chan0 > /sys/module/dmatest/parameters/channel
+ % echo dma0chan1 > /sys/module/dmatest/parameters/channel
+ % echo dma0chan2 > /sys/module/dmatest/parameters/channel
+ % echo 1 > /sys/module/dmatest/parameters/run
+Note: the channel parameter should always be the last parameter set prior to
+running the test (setting run=1), this is because upon setting the channel
+parameter, that specific channel is requested using the dmaengine and a thread
+is created with the existing parameters. This thread is set as pending
+and will be executed once run is set to 1. Any parameters set after the thread
+is created are not applied.
.. hint::
available channel list could be extracted by running the following command::
% ls -1 /sys/class/dma/
-Once started a message like "dmatest: Started 1 threads using dma0chan0" is
-emitted. After that only test failure messages are reported until the test
-stops.
+Once started a message like " dmatest: Added 1 threads using dma0chan0" is
+emitted. A thread for that specific channel is created and is now pending, the
+pending thread is started once run is to 1.
Note that running a new test will not stop any in progress test.
@@ -112,3 +127,85 @@ Example::
The details of a data miscompare error are also emitted, but do not follow the
above format.
+
+Part 5 - Handling channel allocation
+====================================
+
+Allocating Channels
+-------------------
+
+Channels are required to be configured prior to starting the test run.
+Attempting to run the test without configuring the channels will fail.
+
+Example::
+
+ % echo 1 > /sys/module/dmatest/parameters/run
+ dmatest: Could not start test, no channels configured
+
+Channels are registered using the "channel" parameter. Channels can be requested by their
+name, once requested, the channel is registered and a pending thread is added to the test list.
+
+Example::
+
+ % echo dma0chan2 > /sys/module/dmatest/parameters/channel
+ dmatest: Added 1 threads using dma0chan2
+
+More channels can be added by repeating the example above.
+Reading back the channel parameter will return the name of last channel that was added successfully.
+
+Example::
+
+ % echo dma0chan1 > /sys/module/dmatest/parameters/channel
+ dmatest: Added 1 threads using dma0chan1
+ % echo dma0chan2 > /sys/module/dmatest/parameters/channel
+ dmatest: Added 1 threads using dma0chan2
+ % cat /sys/module/dmatest/parameters/channel
+ dma0chan2
+
+Another method of requesting channels is to request a channel with an empty string, Doing so
+will request all channels available to be tested:
+
+Example::
+
+ % echo "" > /sys/module/dmatest/parameters/channel
+ dmatest: Added 1 threads using dma0chan0
+ dmatest: Added 1 threads using dma0chan3
+ dmatest: Added 1 threads using dma0chan4
+ dmatest: Added 1 threads using dma0chan5
+ dmatest: Added 1 threads using dma0chan6
+ dmatest: Added 1 threads using dma0chan7
+ dmatest: Added 1 threads using dma0chan8
+
+At any point during the test configuration, reading the "test_list" parameter will
+print the list of currently pending tests.
+
+Example::
+
+ % cat /sys/module/dmatest/parameters/test_list
+ dmatest: 1 threads using dma0chan0
+ dmatest: 1 threads using dma0chan3
+ dmatest: 1 threads using dma0chan4
+ dmatest: 1 threads using dma0chan5
+ dmatest: 1 threads using dma0chan6
+ dmatest: 1 threads using dma0chan7
+ dmatest: 1 threads using dma0chan8
+
+Note: Channels will have to be configured for each test run as channel configurations do not
+carry across to the next test run.
+
+Releasing Channels
+-------------------
+
+Channels can be freed by setting run to 0.
+
+Example::
+ % echo dma0chan1 > /sys/module/dmatest/parameters/channel
+ dmatest: Added 1 threads using dma0chan1
+ % cat /sys/class/dma/dma0chan1/in_use
+ 1
+ % echo 0 > /sys/module/dmatest/parameters/run
+ % cat /sys/class/dma/dma0chan1/in_use
+ 0
+
+Channels allocated by previous test runs are automatically freed when a new
+channel is requested after completing a successful test run.
^ permalink raw reply related
* [1/4] dt-bindings: dma: uart: add uart dma bindings
From: Long Cheng @ 2018-09-20 6:41 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland
Cc: Matthias Brugger, Dan Williams, Greg Kroah-Hartman, Jiri Slaby,
Ed Blake, Long Cheng, dmaengine, devicetree, linux-arm-kernel,
linux-mediatek, linux-kernel, linux-serial, srv_heupstream,
Yingjoe Chen, YT Shen
add uart dma bindings
Signed-off-by: Long Cheng <long.cheng@mediatek.com>
---
.../devicetree/bindings/dma/8250_mtk_dma.txt | 32 ++++++++++++++++++++
1 file changed, 32 insertions(+)
create mode 100644 Documentation/devicetree/bindings/dma/8250_mtk_dma.txt
diff --git a/Documentation/devicetree/bindings/dma/8250_mtk_dma.txt b/Documentation/devicetree/bindings/dma/8250_mtk_dma.txt
new file mode 100644
index 0000000..b140cf4
--- /dev/null
+++ b/Documentation/devicetree/bindings/dma/8250_mtk_dma.txt
@@ -0,0 +1,32 @@
+* Mediatek UART APDMA Controller
+
+Required properties:
+- compatible should contain:
+ * "mediatek,mt2712-uart-dma" for MT2712 compatible APDMA
+ * "mediatek,mt6577-uart-dma" for MT6577 and all of the above
+
+- reg: The base address of the APDMA register bank.
+
+- interrupts: A single interrupt specifier.
+
+- clocks : Must contain an entry for each entry in clock-names.
+ See ../clocks/clock-bindings.txt for details.
+- clock-names: The APDMA clock for register accesses
+
+Examples:
+
+ apdma: dma-controller@11000380 {
+ compatible = "mediatek,mt2712-uart-dma";
+ reg = <0 0x11000380 0 0x400>;
+ interrupts = <GIC_SPI 63 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 64 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 65 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 66 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 67 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 68 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 69 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 70 IRQ_TYPE_LEVEL_LOW>;
+ clocks = <&pericfg CLK_PERI_AP_DMA>;
+ clock-names = "apdma";
+ #dma-cells = <1>;
+ };
\ No newline at end of file
^ permalink raw reply related
* [2/4] dmaengine: mtk_uart_dma: add Mediatek uart DMA support
From: Long Cheng @ 2018-09-20 6:41 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland
Cc: Matthias Brugger, Dan Williams, Greg Kroah-Hartman, Jiri Slaby,
Ed Blake, Long Cheng, dmaengine, devicetree, linux-arm-kernel,
linux-mediatek, linux-kernel, linux-serial, srv_heupstream,
Yingjoe Chen, YT Shen
In DMA engine framework, add 8250 mtk dma to support it.
Signed-off-by: Long Cheng <long.cheng@mediatek.com>
---
drivers/dma/8250_mtk_dma.c | 1049 ++++++++++++++++++++++++++++++++++++++++++++
drivers/dma/Kconfig | 11 +
drivers/dma/Makefile | 1 +
3 files changed, 1061 insertions(+)
create mode 100644 drivers/dma/8250_mtk_dma.c
diff --git a/drivers/dma/8250_mtk_dma.c b/drivers/dma/8250_mtk_dma.c
new file mode 100644
index 0000000..a07844e
--- /dev/null
+++ b/drivers/dma/8250_mtk_dma.c
@@ -0,0 +1,1049 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Mediatek 8250 DMA driver.
+ *
+ * Copyright (c) 2018 MediaTek Inc.
+ * Author: Long Cheng <long.cheng@mediatek.com>
+ */
+
+#define pr_fmt(fmt) "8250-mtk-dma: " fmt
+#define DRV_NAME "8250-mtk-dma"
+
+#include <linux/clk.h>
+#include <linux/delay.h>
+#include <linux/dmaengine.h>
+#include <linux/dma-mapping.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/of_dma.h>
+#include <linux/of_device.h>
+#include <linux/platform_device.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/pm_runtime.h>
+
+#include "virt-dma.h"
+
+#define MTK_SDMA_REQUESTS 127
+#define MTK_SDMA_CHANNELS (CONFIG_SERIAL_8250_NR_UARTS * 2)
+
+#define VFF_RX_INT_FLAG_CLR_B (BIT(0U) | BIT(1U))
+#define VFF_TX_INT_FLAG_CLR_B 0
+#define VFF_RX_INT_EN0_B BIT(0U) /*rx valid size >= vff thre */
+#define VFF_RX_INT_EN1_B BIT(1U)
+#define VFF_TX_INT_EN_B BIT(0U) /*tx left size >= vff thre */
+#define VFF_INT_EN_CLR_B 0
+#define VFF_WARM_RST_B BIT(0U)
+#define VFF_EN_B BIT(0U)
+#define VFF_STOP_B BIT(0U)
+#define VFF_STOP_CLR_B 0
+#define VFF_FLUSH_B BIT(0U)
+#define VFF_FLUSH_CLR_B 0
+#define VFF_4G_SUPPORT_B BIT(0U)
+#define VFF_4G_SUPPORT_CLR_B 0
+
+/* interrupt trigger level for tx */
+#define VFF_TX_THRE(n) ((n) * 7 / 8)
+/* interrupt trigger level for rx */
+#define VFF_RX_THRE(n) ((n) * 3 / 4)
+
+#define MTK_DMA_RING_SIZE 0xffffU
+/* invert this bit when wrap ring head again*/
+#define MTK_DMA_RING_WRAP 0x10000U
+
+struct mtk_dmadev {
+ struct dma_device ddev;
+ void __iomem *mem_base[MTK_SDMA_CHANNELS];
+ spinlock_t lock; /* dma dev lock */
+ struct tasklet_struct task;
+ struct list_head pending;
+ struct clk *clk;
+ unsigned int dma_requests;
+ bool support_33bits;
+ unsigned int dma_irq[MTK_SDMA_CHANNELS];
+ struct mtk_chan *lch_map[MTK_SDMA_CHANNELS];
+};
+
+struct mtk_chan {
+ struct virt_dma_chan vc;
+ struct list_head node;
+ struct dma_slave_config cfg;
+ void __iomem *channel_base;
+ struct mtk_dma_desc *desc;
+
+ bool paused;
+ bool requested;
+
+ unsigned int dma_sig;
+ unsigned int dma_ch;
+ unsigned int sgidx;
+ unsigned int remain_size;
+ unsigned int rx_ptr;
+
+ /*sync*/
+ struct completion done; /* dma transfer done */
+ spinlock_t lock; /* channel lock */
+ atomic_t loopcnt;
+ atomic_t entry; /* entry count */
+};
+
+struct mtk_dma_sg {
+ dma_addr_t addr;
+ unsigned int en; /* number of elements (24-bit) */
+ unsigned int fn; /* number of frames (16-bit) */
+};
+
+struct mtk_dma_desc {
+ struct virt_dma_desc vd;
+ enum dma_transfer_direction dir;
+ dma_addr_t dev_addr;
+
+ unsigned int sglen;
+ struct mtk_dma_sg sg[0];
+};
+
+enum {
+ VFF_INT_FLAG = 0x00,
+ VFF_INT_EN = 0x04,
+ VFF_EN = 0x08,
+ VFF_RST = 0x0c,
+ VFF_STOP = 0x10,
+ VFF_FLUSH = 0x14,
+ VFF_ADDR = 0x1c,
+ VFF_LEN = 0x24,
+ VFF_THRE = 0x28,
+ VFF_WPT = 0x2c,
+ VFF_RPT = 0x30,
+ /*TX: the buffer size HW can read. RX: the buffer size SW can read.*/
+ VFF_VALID_SIZE = 0x3c,
+ /*TX: the buffer size SW can write. RX: the buffer size HW can write.*/
+ VFF_LEFT_SIZE = 0x40,
+ VFF_DEBUG_STATUS = 0x50,
+ VFF_4G_SUPPORT = 0x54,
+};
+
+static bool mtk_dma_filter_fn(struct dma_chan *chan, void *param);
+static struct of_dma_filter_info mtk_dma_info = {
+ .filter_fn = mtk_dma_filter_fn,
+};
+
+static inline struct mtk_dmadev *to_mtk_dma_dev(struct dma_device *d)
+{
+ return container_of(d, struct mtk_dmadev, ddev);
+}
+
+static inline struct mtk_chan *to_mtk_dma_chan(struct dma_chan *c)
+{
+ return container_of(c, struct mtk_chan, vc.chan);
+}
+
+static inline struct mtk_dma_desc *to_mtk_dma_desc
+ (struct dma_async_tx_descriptor *t)
+{
+ return container_of(t, struct mtk_dma_desc, vd.tx);
+}
+
+static void mtk_dma_chan_write(struct mtk_chan *c,
+ unsigned int reg, unsigned int val)
+{
+ writel(val, c->channel_base + reg);
+}
+
+static unsigned int mtk_dma_chan_read(struct mtk_chan *c, unsigned int reg)
+{
+ return readl(c->channel_base + reg);
+}
+
+static void mtk_dma_desc_free(struct virt_dma_desc *vd)
+{
+ struct dma_chan *chan = vd->tx.chan;
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ unsigned long flags;
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ if (c->desc && c->cfg.direction == DMA_DEV_TO_MEM)
+ atomic_dec(&c->entry);
+
+ kfree(c->desc);
+ c->desc = NULL;
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+}
+
+static int mtk_dma_clk_enable(struct mtk_dmadev *mtkd)
+{
+ int ret;
+
+ ret = clk_prepare_enable(mtkd->clk);
+ if (ret) {
+ dev_err(mtkd->ddev.dev, "Couldn't enable the clock\n");
+ return ret;
+ }
+
+ return 0;
+}
+
+static void mtk_dma_clk_disable(struct mtk_dmadev *mtkd)
+{
+ clk_disable_unprepare(mtkd->clk);
+}
+
+static void mtk_dma_remove_virt_list(dma_cookie_t cookie,
+ struct virt_dma_chan *vc)
+{
+ struct virt_dma_desc *vd;
+
+ if (list_empty(&vc->desc_issued) == 0) {
+ list_for_each_entry(vd, &vc->desc_issued, node) {
+ if (cookie == vd->tx.cookie) {
+ INIT_LIST_HEAD(&vc->desc_issued);
+ break;
+ }
+ }
+ }
+}
+
+static void mtk_dma_tx_flush(struct dma_chan *chan)
+{
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+
+ if (mtk_dma_chan_read(c, VFF_FLUSH) == 0U) {
+ mtk_dma_chan_write(c, VFF_FLUSH, VFF_FLUSH_B);
+ if (atomic_dec_and_test(&c->loopcnt))
+ complete(&c->done);
+ }
+}
+
+/*
+ * check whether the dma flush operation is finished or not.
+ * return 0 for flush success.
+ * return others for flush timeout.
+ */
+static int mtk_dma_check_flush_result(struct dma_chan *chan)
+{
+ struct timespec start, end;
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+
+ start = ktime_to_timespec(ktime_get());
+
+ while ((mtk_dma_chan_read(c, VFF_FLUSH) & VFF_FLUSH_B) == VFF_FLUSH_B) {
+ end = ktime_to_timespec(ktime_get());
+ if ((end.tv_sec - start.tv_sec) > 1 ||
+ ((end.tv_sec - start.tv_sec) == 1 &&
+ end.tv_nsec > start.tv_nsec)) {
+ dev_err(chan->device->dev,
+ "[DMA] Polling flush timeout\n");
+ return -1;
+ }
+ }
+
+ return 0;
+}
+
+static void mtk_dma_tx_write(struct dma_chan *chan)
+{
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+ unsigned int txcount = c->remain_size;
+ unsigned int len, send, left, wpt, wrap;
+
+ if (atomic_inc_return(&c->entry) > 1) {
+ if (vchan_issue_pending(&c->vc) && !c->desc) {
+ spin_lock(&mtkd->lock);
+ list_add_tail(&c->node, &mtkd->pending);
+ spin_unlock(&mtkd->lock);
+ tasklet_schedule(&mtkd->task);
+ }
+ } else {
+ len = mtk_dma_chan_read(c, VFF_LEN);
+ if (mtk_dma_check_flush_result(chan) != 0)
+ return;
+
+ while ((left = mtk_dma_chan_read(c, VFF_LEFT_SIZE)) > 0U) {
+ send = min(left, c->remain_size);
+ wpt = mtk_dma_chan_read(c, VFF_WPT);
+ wrap = wpt & MTK_DMA_RING_WRAP ? 0U : MTK_DMA_RING_WRAP;
+
+ if ((wpt & (len - 1U)) + send < len)
+ mtk_dma_chan_write(c, VFF_WPT, wpt + send);
+ else
+ mtk_dma_chan_write(c, VFF_WPT,
+ ((wpt + send) & (len - 1U))
+ | wrap);
+
+ c->remain_size -= send;
+ if (c->remain_size == 0U)
+ break;
+ }
+
+ if (txcount != c->remain_size) {
+ mtk_dma_chan_write(c, VFF_INT_EN, VFF_TX_INT_EN_B);
+ mtk_dma_tx_flush(chan);
+ }
+ }
+ atomic_dec(&c->entry);
+}
+
+static void mtk_dma_start_tx(struct mtk_chan *c)
+{
+ if (mtk_dma_chan_read(c, VFF_LEFT_SIZE) == 0U) {
+ pr_info("%s maybe need fix? @L %d\n", __func__, __LINE__);
+ mtk_dma_chan_write(c, VFF_INT_EN, VFF_TX_INT_EN_B);
+ } else {
+ reinit_completion(&c->done);
+
+ /* inc twice, once for tx_flush, another for tx_interrupt */
+ atomic_inc(&c->loopcnt);
+ atomic_inc(&c->loopcnt);
+ mtk_dma_tx_write(&c->vc.chan);
+ }
+ c->paused = false;
+}
+
+static void mtk_dma_get_rx_size(struct mtk_chan *c)
+{
+ unsigned int count;
+ unsigned int rdptr, wrptr, wrreg, rdreg;
+ unsigned int rx_size = mtk_dma_chan_read(c, VFF_LEN);
+
+ rdreg = mtk_dma_chan_read(c, VFF_RPT);
+ wrreg = mtk_dma_chan_read(c, VFF_WPT);
+ rdptr = rdreg & MTK_DMA_RING_SIZE;
+ wrptr = wrreg & MTK_DMA_RING_SIZE;
+ count = ((rdreg ^ wrreg) & MTK_DMA_RING_WRAP) ?
+ (wrptr + rx_size - rdptr) : (wrptr - rdptr);
+
+ c->remain_size = count;
+ c->rx_ptr = rdptr;
+
+ mtk_dma_chan_write(c, VFF_RPT, wrreg);
+}
+
+static void mtk_dma_start_rx(struct mtk_chan *c)
+{
+ struct dma_chan *chan = &c->vc.chan;
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+ struct mtk_dma_desc *d = c->desc;
+
+ if (mtk_dma_chan_read(c, VFF_VALID_SIZE) != 0U &&
+ d && d->vd.tx.cookie != 0) {
+ mtk_dma_get_rx_size(c);
+ mtk_dma_remove_virt_list(d->vd.tx.cookie, &c->vc);
+ vchan_cookie_complete(&d->vd);
+ } else {
+ if (mtk_dma_chan_read(c, VFF_VALID_SIZE) != 0U) {
+ spin_lock(&mtkd->lock);
+ if (list_empty(&mtkd->pending))
+ list_add_tail(&c->node, &mtkd->pending);
+ spin_unlock(&mtkd->lock);
+ tasklet_schedule(&mtkd->task);
+ } else {
+ if (atomic_read(&c->entry) > 0)
+ atomic_set(&c->entry, 0);
+ }
+ }
+}
+
+static void mtk_dma_reset(struct mtk_chan *c)
+{
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(c->vc.chan.device);
+
+ mtk_dma_chan_write(c, VFF_ADDR, 0);
+ mtk_dma_chan_write(c, VFF_THRE, 0);
+ mtk_dma_chan_write(c, VFF_LEN, 0);
+ mtk_dma_chan_write(c, VFF_RST, VFF_WARM_RST_B);
+
+ while
+ (mtk_dma_chan_read(c, VFF_EN));
+
+ if (c->cfg.direction == DMA_DEV_TO_MEM)
+ mtk_dma_chan_write(c, VFF_RPT, 0);
+ else if (c->cfg.direction == DMA_MEM_TO_DEV)
+ mtk_dma_chan_write(c, VFF_WPT, 0);
+ else
+ dev_info(c->vc.chan.device->dev, "Unknown direction.\n");
+
+ if (mtkd->support_33bits)
+ mtk_dma_chan_write(c, VFF_4G_SUPPORT, VFF_4G_SUPPORT_CLR_B);
+}
+
+static void mtk_dma_stop(struct mtk_chan *c)
+{
+ int polling_cnt;
+
+ mtk_dma_chan_write(c, VFF_FLUSH, VFF_FLUSH_CLR_B);
+
+ polling_cnt = 0;
+ while ((mtk_dma_chan_read(c, VFF_FLUSH) & VFF_FLUSH_B) ==
+ VFF_FLUSH_B) {
+ polling_cnt++;
+ if (polling_cnt > 10000) {
+ dev_err(c->vc.chan.device->dev,
+ "dma stop: polling FLUSH fail, DEBUG=0x%x\n",
+ mtk_dma_chan_read(c, VFF_DEBUG_STATUS));
+ break;
+ }
+ }
+
+ polling_cnt = 0;
+ /*set stop as 1 -> wait until en is 0 -> set stop as 0*/
+ mtk_dma_chan_write(c, VFF_STOP, VFF_STOP_B);
+ while (mtk_dma_chan_read(c, VFF_EN)) {
+ polling_cnt++;
+ if (polling_cnt > 10000) {
+ dev_err(c->vc.chan.device->dev,
+ "dma stop: polling VFF_EN fail, DEBUG=0x%x\n",
+ mtk_dma_chan_read(c, VFF_DEBUG_STATUS));
+ break;
+ }
+ }
+ mtk_dma_chan_write(c, VFF_STOP, VFF_STOP_CLR_B);
+ mtk_dma_chan_write(c, VFF_INT_EN, VFF_INT_EN_CLR_B);
+
+ if (c->cfg.direction == DMA_DEV_TO_MEM)
+ mtk_dma_chan_write(c, VFF_INT_FLAG, VFF_RX_INT_FLAG_CLR_B);
+ else
+ mtk_dma_chan_write(c, VFF_INT_FLAG, VFF_TX_INT_FLAG_CLR_B);
+
+ c->paused = true;
+}
+
+/*
+ * We need to deal with 'all channels in-use'
+ */
+static void mtk_dma_rx_sched(struct mtk_chan *c)
+{
+ struct dma_chan *chan = &c->vc.chan;
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+
+ if (atomic_read(&c->entry) < 1) {
+ mtk_dma_start_rx(c);
+ } else {
+ spin_lock(&mtkd->lock);
+ if (list_empty(&mtkd->pending))
+ list_add_tail(&c->node, &mtkd->pending);
+ spin_unlock(&mtkd->lock);
+ tasklet_schedule(&mtkd->task);
+ }
+}
+
+/*
+ * This callback schedules all pending channels. We could be more
+ * clever here by postponing allocation of the real DMA channels to
+ * this point, and freeing them when our virtual channel becomes idle.
+ *
+ * We would then need to deal with 'all channels in-use'
+ */
+static void mtk_dma_sched(unsigned long data)
+{
+ struct mtk_dmadev *mtkd = (struct mtk_dmadev *)data;
+ struct mtk_chan *c;
+ struct virt_dma_desc *vd;
+ dma_cookie_t cookie;
+ LIST_HEAD(head);
+ unsigned long flags;
+
+ spin_lock_irq(&mtkd->lock);
+ list_splice_tail_init(&mtkd->pending, &head);
+ spin_unlock_irq(&mtkd->lock);
+
+ if (list_empty(&head) == 0) {
+ c = list_first_entry(&head, struct mtk_chan, node);
+ cookie = c->vc.chan.cookie;
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ if (c->cfg.direction == DMA_DEV_TO_MEM) {
+ list_del_init(&c->node);
+ mtk_dma_rx_sched(c);
+ } else if (c->cfg.direction == DMA_MEM_TO_DEV) {
+ vd = vchan_find_desc(&c->vc, cookie);
+
+ c->desc = to_mtk_dma_desc(&vd->tx);
+ list_del_init(&c->node);
+ mtk_dma_start_tx(c);
+ }
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+ }
+}
+
+static int mtk_dma_alloc_chan_resources(struct dma_chan *chan)
+{
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ int ret = -EBUSY;
+
+ pm_runtime_get_sync(mtkd->ddev.dev);
+
+ if (!mtkd->lch_map[c->dma_ch]) {
+ c->channel_base = mtkd->mem_base[c->dma_ch];
+ mtkd->lch_map[c->dma_ch] = c;
+ ret = 1;
+ }
+ c->requested = false;
+ mtk_dma_reset(c);
+
+ return ret;
+}
+
+static void mtk_dma_free_chan_resources(struct dma_chan *chan)
+{
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+
+ if (c->requested) {
+ c->requested = false;
+ free_irq(mtkd->dma_irq[c->dma_ch], chan);
+ }
+
+ tasklet_kill(&mtkd->task);
+
+ c->channel_base = NULL;
+ mtkd->lch_map[c->dma_ch] = NULL;
+ vchan_free_chan_resources(&c->vc);
+
+ dev_dbg(mtkd->ddev.dev, "freeing channel for %u\n", c->dma_sig);
+ c->dma_sig = 0;
+
+ pm_runtime_put_sync(mtkd->ddev.dev);
+}
+
+static enum dma_status mtk_dma_tx_status(struct dma_chan *chan,
+ dma_cookie_t cookie,
+ struct dma_tx_state *txstate)
+{
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ enum dma_status ret;
+ unsigned long flags;
+
+ ret = dma_cookie_status(chan, cookie, txstate);
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ if (ret == DMA_IN_PROGRESS) {
+ c->rx_ptr = mtk_dma_chan_read(c, VFF_RPT) & MTK_DMA_RING_SIZE;
+ txstate->residue = c->rx_ptr;
+ } else if (ret == DMA_COMPLETE && c->cfg.direction == DMA_DEV_TO_MEM) {
+ txstate->residue = c->remain_size;
+ } else {
+ txstate->residue = 0;
+ }
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+
+ return ret;
+}
+
+static unsigned int mtk_dma_desc_size(struct mtk_dma_desc *d)
+{
+ struct mtk_dma_sg *sg;
+ unsigned int i;
+ unsigned int size;
+
+ for (size = i = 0; i < d->sglen; i++) {
+ sg = &d->sg[i];
+ size += sg->en * sg->fn;
+ }
+ return size;
+}
+
+static struct dma_async_tx_descriptor *mtk_dma_prep_slave_sg
+ (struct dma_chan *chan, struct scatterlist *sgl,
+ unsigned int sglen, enum dma_transfer_direction dir,
+ unsigned long tx_flags, void *context)
+{
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ struct scatterlist *sgent;
+ struct mtk_dma_desc *d;
+ dma_addr_t dev_addr;
+ unsigned int i, j, en, frame_bytes;
+
+ en = 1;
+ frame_bytes = 1;
+
+ if (dir == DMA_DEV_TO_MEM) {
+ dev_addr = c->cfg.src_addr;
+ } else if (dir == DMA_MEM_TO_DEV) {
+ dev_addr = c->cfg.dst_addr;
+ } else {
+ dev_err(chan->device->dev, "bad direction\n");
+ return NULL;
+ }
+
+ /* Now allocate and setup the descriptor. */
+ d = kzalloc(sizeof(*d) + sglen * sizeof(d->sg[0]), GFP_ATOMIC);
+ if (!d)
+ return NULL;
+
+ d->dir = dir;
+ d->dev_addr = dev_addr;
+
+ j = 0;
+ for_each_sg(sgl, sgent, sglen, i) {
+ d->sg[j].addr = sg_dma_address(sgent);
+ d->sg[j].en = en;
+ d->sg[j].fn = sg_dma_len(sgent) / frame_bytes;
+ j++;
+ }
+
+ d->sglen = j;
+
+ if (dir == DMA_MEM_TO_DEV)
+ c->remain_size = mtk_dma_desc_size(d);
+
+ return vchan_tx_prep(&c->vc, &d->vd, tx_flags);
+}
+
+static void mtk_dma_issue_pending(struct dma_chan *chan)
+{
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ struct mtk_dmadev *mtkd;
+ struct virt_dma_desc *vd;
+ dma_cookie_t cookie;
+ unsigned long flags;
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ if (c->cfg.direction == DMA_DEV_TO_MEM) {
+ cookie = c->vc.chan.cookie;
+ mtkd = to_mtk_dma_dev(chan->device);
+ if (vchan_issue_pending(&c->vc) && !c->desc) {
+ vd = vchan_find_desc(&c->vc, cookie);
+ c->desc = to_mtk_dma_desc(&vd->tx);
+ if (atomic_read(&c->entry) > 0)
+ atomic_set(&c->entry, 0);
+ }
+ } else if (c->cfg.direction == DMA_MEM_TO_DEV) {
+ cookie = c->vc.chan.cookie;
+ if (vchan_issue_pending(&c->vc) && !c->desc) {
+ vd = vchan_find_desc(&c->vc, cookie);
+ c->desc = to_mtk_dma_desc(&vd->tx);
+ mtk_dma_start_tx(c);
+ }
+ }
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+}
+
+static irqreturn_t mtk_dma_rx_interrupt(int irq, void *dev_id)
+{
+ struct dma_chan *chan = (struct dma_chan *)dev_id;
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+ unsigned long flags;
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ mtk_dma_chan_write(c, VFF_INT_FLAG, VFF_RX_INT_FLAG_CLR_B);
+
+ if (atomic_inc_return(&c->entry) > 1) {
+ if (list_empty(&mtkd->pending))
+ list_add_tail(&c->node, &mtkd->pending);
+ tasklet_schedule(&mtkd->task);
+ } else {
+ mtk_dma_start_rx(c);
+ }
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t mtk_dma_tx_interrupt(int irq, void *dev_id)
+{
+ struct dma_chan *chan = (struct dma_chan *)dev_id;
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+ struct mtk_dma_desc *d = c->desc;
+ unsigned long flags;
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ if (c->remain_size != 0U) {
+ list_add_tail(&c->node, &mtkd->pending);
+ tasklet_schedule(&mtkd->task);
+ } else {
+ mtk_dma_remove_virt_list(d->vd.tx.cookie, &c->vc);
+ vchan_cookie_complete(&d->vd);
+ }
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+
+ mtk_dma_chan_write(c, VFF_INT_FLAG, VFF_TX_INT_FLAG_CLR_B);
+ if (atomic_dec_and_test(&c->loopcnt))
+ complete(&c->done);
+
+ return IRQ_HANDLED;
+}
+
+static int mtk_dma_slave_config(struct dma_chan *chan,
+ struct dma_slave_config *cfg)
+{
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(c->vc.chan.device);
+ int ret;
+
+ c->cfg = *cfg;
+
+ if (cfg->direction == DMA_DEV_TO_MEM) {
+ unsigned int rx_len = cfg->src_addr_width * 1024;
+
+ mtk_dma_chan_write(c, VFF_ADDR, cfg->src_addr);
+ mtk_dma_chan_write(c, VFF_LEN, rx_len);
+ mtk_dma_chan_write(c, VFF_THRE, VFF_RX_THRE(rx_len));
+ mtk_dma_chan_write(c,
+ VFF_INT_EN, VFF_RX_INT_EN0_B
+ | VFF_RX_INT_EN1_B);
+ mtk_dma_chan_write(c, VFF_INT_FLAG, VFF_RX_INT_FLAG_CLR_B);
+ mtk_dma_chan_write(c, VFF_EN, VFF_EN_B);
+
+ if (!c->requested) {
+ atomic_set(&c->entry, 0);
+ c->requested = true;
+ ret = request_irq(mtkd->dma_irq[c->dma_ch],
+ mtk_dma_rx_interrupt,
+ IRQF_TRIGGER_NONE,
+ DRV_NAME, chan);
+ if (ret < 0) {
+ dev_err(chan->device->dev, "Can't request rx dma IRQ\n");
+ return -EINVAL;
+ }
+ }
+ } else if (cfg->direction == DMA_MEM_TO_DEV) {
+ unsigned int tx_len = cfg->dst_addr_width * 1024;
+
+ mtk_dma_chan_write(c, VFF_ADDR, cfg->dst_addr);
+ mtk_dma_chan_write(c, VFF_LEN, tx_len);
+ mtk_dma_chan_write(c, VFF_THRE, VFF_TX_THRE(tx_len));
+ mtk_dma_chan_write(c, VFF_INT_FLAG, VFF_TX_INT_FLAG_CLR_B);
+ mtk_dma_chan_write(c, VFF_EN, VFF_EN_B);
+
+ if (!c->requested) {
+ c->requested = true;
+ ret = request_irq(mtkd->dma_irq[c->dma_ch],
+ mtk_dma_tx_interrupt,
+ IRQF_TRIGGER_NONE,
+ DRV_NAME, chan);
+ if (ret < 0) {
+ dev_err(chan->device->dev, "Can't request tx dma IRQ\n");
+ return -EINVAL;
+ }
+ }
+ } else {
+ dev_info(chan->device->dev, "Unknown direction!\n");
+ }
+
+ if (mtkd->support_33bits)
+ mtk_dma_chan_write(c, VFF_4G_SUPPORT, VFF_4G_SUPPORT_B);
+
+ if (mtk_dma_chan_read(c, VFF_EN) != VFF_EN_B) {
+ dev_err(chan->device->dev,
+ "config dma dir[%d] fail\n", cfg->direction);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int mtk_dma_terminate_all(struct dma_chan *chan)
+{
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ unsigned long flags;
+ LIST_HEAD(head);
+
+ if (atomic_read(&c->loopcnt) != 0)
+ wait_for_completion(&c->done);
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ if (c->desc) {
+ mtk_dma_remove_virt_list(c->desc->vd.tx.cookie, &c->vc);
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+
+ mtk_dma_desc_free(&c->desc->vd);
+
+ spin_lock_irqsave(&c->vc.lock, flags);
+ if (!c->paused) {
+ list_del_init(&c->node);
+ mtk_dma_stop(c);
+ }
+ }
+ vchan_get_all_descriptors(&c->vc, &head);
+ spin_unlock_irqrestore(&c->vc.lock, flags);
+
+ vchan_dma_desc_free_list(&c->vc, &head);
+
+ return 0;
+}
+
+static int mtk_dma_device_pause(struct dma_chan *chan)
+{
+ /* Pause/Resume only allowed with cyclic mode */
+ return -EINVAL;
+}
+
+static int mtk_dma_device_resume(struct dma_chan *chan)
+{
+ /* Pause/Resume only allowed with cyclic mode */
+ return -EINVAL;
+}
+
+static int mtk_dma_chan_init(struct mtk_dmadev *mtkd)
+{
+ struct mtk_chan *c;
+
+ c = devm_kzalloc(mtkd->ddev.dev, sizeof(*c), GFP_KERNEL);
+ if (!c)
+ return -ENOMEM;
+
+ c->vc.desc_free = mtk_dma_desc_free;
+ vchan_init(&c->vc, &mtkd->ddev);
+ spin_lock_init(&c->lock);
+ INIT_LIST_HEAD(&c->node);
+
+ init_completion(&c->done);
+ atomic_set(&c->loopcnt, 0);
+ atomic_set(&c->entry, 0);
+
+ return 0;
+}
+
+static void mtk_dma_free(struct mtk_dmadev *mtkd)
+{
+ tasklet_kill(&mtkd->task);
+ while (list_empty(&mtkd->ddev.channels) == 0) {
+ struct mtk_chan *c = list_first_entry(&mtkd->ddev.channels,
+ struct mtk_chan, vc.chan.device_node);
+
+ list_del(&c->vc.chan.device_node);
+ tasklet_kill(&c->vc.task);
+ devm_kfree(mtkd->ddev.dev, c);
+ }
+}
+
+static const struct of_device_id mtk_uart_dma_match[] = {
+ { .compatible = "mediatek,mt6577-uart-dma", },
+ { /* sentinel */ },
+};
+MODULE_DEVICE_TABLE(of, mtk_uart_dma_match);
+
+static int mtk_dma_probe(struct platform_device *pdev)
+{
+ struct mtk_dmadev *mtkd;
+ struct resource *res;
+ unsigned int i;
+ int rc;
+
+ mtkd = devm_kzalloc(&pdev->dev, sizeof(*mtkd), GFP_KERNEL);
+ if (!mtkd)
+ return -ENOMEM;
+
+ for (i = 0; i < MTK_SDMA_CHANNELS; i++) {
+ res = platform_get_resource(pdev, IORESOURCE_MEM, i);
+ if (!res)
+ return -ENODEV;
+ mtkd->mem_base[i] = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(mtkd->mem_base[i]))
+ return PTR_ERR(mtkd->mem_base[i]);
+ }
+
+ /* request irq */
+ for (i = 0; i < MTK_SDMA_CHANNELS; i++) {
+ mtkd->dma_irq[i] = platform_get_irq(pdev, i);
+ if ((int)mtkd->dma_irq[i] < 0) {
+ dev_err(&pdev->dev, "failed to get IRQ[%d]\n", i);
+ return -EINVAL;
+ }
+ }
+
+ mtkd->clk = devm_clk_get(&pdev->dev, NULL);
+ if (IS_ERR(mtkd->clk)) {
+ dev_err(&pdev->dev, "No clock specified\n");
+ return PTR_ERR(mtkd->clk);
+ }
+
+ if (of_property_read_bool(pdev->dev.of_node, "dma-33bits")) {
+ dev_info(&pdev->dev, "Support dma 33bits\n");
+ mtkd->support_33bits = true;
+ }
+
+ if (mtkd->support_33bits)
+ rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(33));
+ else
+ rc = dma_set_mask_and_coherent(&pdev->dev, DMA_BIT_MASK(32));
+ if (rc)
+ return rc;
+
+ dma_cap_set(DMA_SLAVE, mtkd->ddev.cap_mask);
+ mtkd->ddev.device_alloc_chan_resources = mtk_dma_alloc_chan_resources;
+ mtkd->ddev.device_free_chan_resources = mtk_dma_free_chan_resources;
+ mtkd->ddev.device_tx_status = mtk_dma_tx_status;
+ mtkd->ddev.device_issue_pending = mtk_dma_issue_pending;
+ mtkd->ddev.device_prep_slave_sg = mtk_dma_prep_slave_sg;
+ mtkd->ddev.device_config = mtk_dma_slave_config;
+ mtkd->ddev.device_pause = mtk_dma_device_pause;
+ mtkd->ddev.device_resume = mtk_dma_device_resume;
+ mtkd->ddev.device_terminate_all = mtk_dma_terminate_all;
+ mtkd->ddev.src_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_1_BYTE);
+ mtkd->ddev.dst_addr_widths = BIT(DMA_SLAVE_BUSWIDTH_1_BYTE);
+ mtkd->ddev.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV);
+ mtkd->ddev.residue_granularity = DMA_RESIDUE_GRANULARITY_SEGMENT;
+ mtkd->ddev.dev = &pdev->dev;
+ INIT_LIST_HEAD(&mtkd->ddev.channels);
+ INIT_LIST_HEAD(&mtkd->pending);
+
+ spin_lock_init(&mtkd->lock);
+ tasklet_init(&mtkd->task, mtk_dma_sched, (unsigned long)mtkd);
+
+ mtkd->dma_requests = MTK_SDMA_REQUESTS;
+ if (of_property_read_u32(pdev->dev.of_node,
+ "dma-requests", &mtkd->dma_requests) != 0) {
+ dev_info(&pdev->dev,
+ "Missing dma-requests property, using %u.\n",
+ MTK_SDMA_REQUESTS);
+ }
+
+ for (i = 0; i < MTK_SDMA_CHANNELS; i++) {
+ rc = mtk_dma_chan_init(mtkd);
+ if (rc)
+ goto err_no_dma;
+ }
+
+ pm_runtime_enable(&pdev->dev);
+ pm_runtime_set_active(&pdev->dev);
+
+ rc = dma_async_device_register(&mtkd->ddev);
+ if (rc) {
+ dev_warn(&pdev->dev, "fail to register DMA device: %d\n", rc);
+ mtk_dma_clk_disable(mtkd);
+ goto err_no_dma;
+ }
+
+ platform_set_drvdata(pdev, mtkd);
+
+ if (pdev->dev.of_node) {
+ mtk_dma_info.dma_cap = mtkd->ddev.cap_mask;
+
+ /* Device-tree DMA controller registration */
+ rc = of_dma_controller_register(pdev->dev.of_node,
+ of_dma_simple_xlate,
+ &mtk_dma_info);
+ if (rc) {
+ dev_warn(&pdev->dev, "fail to register DMA controller\n");
+ dma_async_device_unregister(&mtkd->ddev);
+ mtk_dma_clk_disable(mtkd);
+ goto err_no_dma;
+ }
+ }
+
+ return rc;
+
+err_no_dma:
+ mtk_dma_free(mtkd);
+ return rc;
+}
+
+static int mtk_dma_remove(struct platform_device *pdev)
+{
+ struct mtk_dmadev *mtkd = platform_get_drvdata(pdev);
+
+ if (pdev->dev.of_node)
+ of_dma_controller_free(pdev->dev.of_node);
+
+ pm_runtime_disable(&pdev->dev);
+ pm_runtime_put_noidle(&pdev->dev);
+
+ dma_async_device_unregister(&mtkd->ddev);
+
+ mtk_dma_free(mtkd);
+
+ return 0;
+}
+
+#ifdef CONFIG_PM_SLEEP
+static int mtk_dma_suspend(struct device *dev)
+{
+ struct mtk_dmadev *mtkd = dev_get_drvdata(dev);
+
+ if (!pm_runtime_suspended(dev))
+ mtk_dma_clk_disable(mtkd);
+
+ return 0;
+}
+
+static int mtk_dma_resume(struct device *dev)
+{
+ int ret;
+ struct mtk_dmadev *mtkd = dev_get_drvdata(dev);
+
+ if (!pm_runtime_suspended(dev)) {
+ ret = mtk_dma_clk_enable(mtkd);
+ if (ret) {
+ dev_info(dev, "fail to enable clk: %d\n", ret);
+ return ret;
+ }
+ }
+
+ return 0;
+}
+
+static int mtk_dma_runtime_suspend(struct device *dev)
+{
+ struct mtk_dmadev *mtkd = dev_get_drvdata(dev);
+
+ mtk_dma_clk_disable(mtkd);
+
+ return 0;
+}
+
+static int mtk_dma_runtime_resume(struct device *dev)
+{
+ int ret;
+ struct mtk_dmadev *mtkd = dev_get_drvdata(dev);
+
+ ret = mtk_dma_clk_enable(mtkd);
+ if (ret) {
+ dev_warn(dev, "fail to enable clk: %d\n", ret);
+ return ret;
+ }
+
+ return 0;
+}
+
+#endif /* CONFIG_PM_SLEEP */
+
+static const struct dev_pm_ops mtk_dma_pm_ops = {
+ SET_SYSTEM_SLEEP_PM_OPS(mtk_dma_suspend, mtk_dma_resume)
+ SET_RUNTIME_PM_OPS(mtk_dma_runtime_suspend,
+ mtk_dma_runtime_resume, NULL)
+};
+
+static struct platform_driver mtk_dma_driver = {
+ .probe = mtk_dma_probe,
+ .remove = mtk_dma_remove,
+ .driver = {
+ .name = "8250-mtk-dma",
+ .pm = &mtk_dma_pm_ops,
+ .of_match_table = of_match_ptr(mtk_uart_dma_match),
+ },
+};
+
+static bool mtk_dma_filter_fn(struct dma_chan *chan, void *param)
+{
+ if (chan->device->dev->driver == &mtk_dma_driver.driver) {
+ struct mtk_dmadev *mtkd = to_mtk_dma_dev(chan->device);
+ struct mtk_chan *c = to_mtk_dma_chan(chan);
+ unsigned int req = *(unsigned int *)param;
+
+ if (req <= mtkd->dma_requests) {
+ c->dma_sig = req;
+ c->dma_ch = req;
+ return true;
+ }
+ }
+ return false;
+}
+
+static int mtk_dma_init(void)
+{
+ return platform_driver_register(&mtk_dma_driver);
+}
+subsys_initcall(mtk_dma_init);
+
+static void __exit mtk_dma_exit(void)
+{
+ platform_driver_unregister(&mtk_dma_driver);
+}
+module_exit(mtk_dma_exit);
diff --git a/drivers/dma/Kconfig b/drivers/dma/Kconfig
index dacf3f4..cfa1699 100644
--- a/drivers/dma/Kconfig
+++ b/drivers/dma/Kconfig
@@ -151,6 +151,17 @@ config DMA_JZ4780
If you have a board based on such a SoC and wish to use DMA for
devices which can use the DMA controller, say Y or M here.
+config DMA_MTK_UART
+ tristate "MediaTek SoCs APDMA support for UART"
+ depends on OF
+ select DMA_ENGINE
+ select DMA_VIRTUAL_CHANNELS
+ help
+ Support for the UART DMA engine found on MediaTek MTK SoCs.
+ when 8250 mtk uart is enabled, and if you want to using DMA,
+ you can enable the config. the DMA engine just only be used
+ with MediaTek Socs.
+
config DMA_SA11X0
tristate "SA-11x0 DMA support"
depends on ARCH_SA1100 || COMPILE_TEST
diff --git a/drivers/dma/Makefile b/drivers/dma/Makefile
index c91702d..42690d8 100644
--- a/drivers/dma/Makefile
+++ b/drivers/dma/Makefile
@@ -24,6 +24,7 @@ obj-$(CONFIG_COH901318) += coh901318.o coh901318_lli.o
obj-$(CONFIG_DMA_BCM2835) += bcm2835-dma.o
obj-$(CONFIG_DMA_JZ4740) += dma-jz4740.o
obj-$(CONFIG_DMA_JZ4780) += dma-jz4780.o
+obj-$(CONFIG_DMA_MTK_UART) += 8250_mtk_dma.o
obj-$(CONFIG_DMA_SA11X0) += sa11x0-dma.o
obj-$(CONFIG_DMA_SUN4I) += sun4i-dma.o
obj-$(CONFIG_DMA_SUN6I) += sun6i-dma.o
^ permalink raw reply related
* [3/4] serial: 8250-mtk: add uart DMA support
From: Long Cheng @ 2018-09-20 6:41 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland
Cc: Matthias Brugger, Dan Williams, Greg Kroah-Hartman, Jiri Slaby,
Ed Blake, Long Cheng, dmaengine, devicetree, linux-arm-kernel,
linux-mediatek, linux-kernel, linux-serial, srv_heupstream,
Yingjoe Chen, YT Shen
Modify uart register to support DMA function.
Signed-off-by: Long Cheng <long.cheng@mediatek.com>
---
drivers/tty/serial/8250/8250_mtk.c | 211 +++++++++++++++++++++++++++++++++++-
1 file changed, 210 insertions(+), 1 deletion(-)
diff --git a/drivers/tty/serial/8250/8250_mtk.c b/drivers/tty/serial/8250/8250_mtk.c
index dd5e1ce..9da9db4 100644
--- a/drivers/tty/serial/8250/8250_mtk.c
+++ b/drivers/tty/serial/8250/8250_mtk.c
@@ -14,6 +14,10 @@
#include <linux/pm_runtime.h>
#include <linux/serial_8250.h>
#include <linux/serial_reg.h>
+#include <linux/console.h>
+#include <linux/dma-mapping.h>
+#include <linux/tty.h>
+#include <linux/tty_flip.h>
#include "8250.h"
@@ -22,12 +26,173 @@
#define UART_MTK_SAMPLE_POINT 0x0b /* Sample point register */
#define MTK_UART_RATE_FIX 0x0d /* UART Rate Fix Register */
+#define MTK_UART_DMA_EN 0x13 /* DMA Enable register */
+#define MTK_UART_DMA_EN_TX 0x2
+#define MTK_UART_DMA_EN_RX 0x5
+
+#define MTK_UART_TX_SIZE UART_XMIT_SIZE
+#define MTK_UART_RX_SIZE 0x8000
+#define MTK_UART_TX_TRIGGER 1
+#define MTK_UART_RX_TRIGGER MTK_UART_RX_SIZE
+
+#ifdef CONFIG_SERIAL_8250_DMA
+enum dma_rx_status {
+ DMA_RX_START = 0,
+ DMA_RX_RUNNING = 1,
+ DMA_RX_SHUTDOWN = 2,
+};
+#endif
+
struct mtk8250_data {
int line;
+ unsigned int rx_pos;
struct clk *uart_clk;
struct clk *bus_clk;
+ struct uart_8250_dma *dma;
+#ifdef CONFIG_SERIAL_8250_DMA
+ enum dma_rx_status rx_status;
+#endif
};
+#ifdef CONFIG_SERIAL_8250_DMA
+static void mtk8250_rx_dma(struct uart_8250_port *up);
+
+static void mtk8250_dma_rx_complete(void *param)
+{
+ struct uart_8250_port *up = param;
+ struct uart_8250_dma *dma = up->dma;
+ struct mtk8250_data *data = up->port.private_data;
+ struct tty_port *tty_port = &up->port.state->port;
+ struct dma_tx_state state;
+ unsigned char *ptr;
+ int copied;
+
+ dma_sync_single_for_cpu(dma->rxchan->device->dev, dma->rx_addr,
+ dma->rx_size, DMA_FROM_DEVICE);
+
+ dmaengine_tx_status(dma->rxchan, dma->rx_cookie, &state);
+ dmaengine_terminate_all(dma->rxchan);
+
+ if (data->rx_status == DMA_RX_SHUTDOWN)
+ return;
+
+ if ((data->rx_pos + state.residue) <= dma->rx_size) {
+ ptr = (unsigned char *)(data->rx_pos + dma->rx_buf);
+ copied = tty_insert_flip_string(tty_port, ptr, state.residue);
+ } else {
+ ptr = (unsigned char *)(data->rx_pos + dma->rx_buf);
+ copied = tty_insert_flip_string(tty_port, ptr,
+ dma->rx_size - data->rx_pos);
+ ptr = (unsigned char *)(dma->rx_buf);
+ copied += tty_insert_flip_string(tty_port, ptr,
+ data->rx_pos + state.residue - dma->rx_size);
+ }
+ up->port.icount.rx += copied;
+
+ tty_flip_buffer_push(tty_port);
+
+ mtk8250_rx_dma(up);
+}
+
+static void mtk8250_rx_dma(struct uart_8250_port *up)
+{
+ struct uart_8250_dma *dma = up->dma;
+ struct mtk8250_data *data = up->port.private_data;
+ struct dma_async_tx_descriptor *desc;
+ struct dma_tx_state state;
+
+ desc = dmaengine_prep_slave_single(dma->rxchan, dma->rx_addr,
+ dma->rx_size, DMA_DEV_TO_MEM,
+ DMA_PREP_INTERRUPT | DMA_CTRL_ACK);
+ if (!desc) {
+ pr_err("failed to prepare rx slave single\n");
+ return;
+ }
+
+ desc->callback = mtk8250_dma_rx_complete;
+ desc->callback_param = up;
+
+ dma->rx_cookie = dmaengine_submit(desc);
+
+ dmaengine_tx_status(dma->rxchan, dma->rx_cookie, &state);
+ data->rx_pos = state.residue;
+
+ dma_sync_single_for_device(dma->rxchan->device->dev, dma->rx_addr,
+ dma->rx_size, DMA_FROM_DEVICE);
+
+ dma_async_issue_pending(dma->rxchan);
+}
+
+static void mtk8250_dma_enable(struct uart_8250_port *up)
+{
+ struct uart_8250_dma *dma = up->dma;
+ struct mtk8250_data *data = up->port.private_data;
+ int lcr = serial_in(up, UART_LCR);
+
+ if (data->rx_status != DMA_RX_START)
+ return;
+
+ dma->rxconf.direction = DMA_DEV_TO_MEM;
+ dma->rxconf.src_addr_width = dma->rx_size / 1024;
+ dma->rxconf.src_addr = dma->rx_addr;
+
+ dma->txconf.direction = DMA_MEM_TO_DEV;
+ dma->txconf.dst_addr_width = MTK_UART_TX_SIZE / 1024;
+ dma->txconf.dst_addr = dma->tx_addr;
+
+ serial_out(up, UART_FCR, UART_FCR_ENABLE_FIFO | UART_FCR_CLEAR_RCVR |
+ UART_FCR_CLEAR_XMIT);
+ serial_out(up, MTK_UART_DMA_EN,
+ MTK_UART_DMA_EN_RX | MTK_UART_DMA_EN_TX);
+
+ serial_out(up, UART_LCR, UART_LCR_CONF_MODE_B);
+ serial_out(up, UART_EFR, UART_EFR_ECB);
+ serial_out(up, UART_LCR, lcr);
+
+ if (dmaengine_slave_config(dma->rxchan, &dma->rxconf) != 0)
+ pr_err("failed to configure rx dma channel\n");
+ if (dmaengine_slave_config(dma->txchan, &dma->txconf) != 0)
+ pr_err("failed to configure tx dma channel\n");
+
+ data->rx_status = DMA_RX_RUNNING;
+ data->rx_pos = 0;
+ mtk8250_rx_dma(up);
+}
+#endif
+
+static int mtk8250_startup(struct uart_port *port)
+{
+#ifdef CONFIG_SERIAL_8250_DMA
+ struct uart_8250_port *up = up_to_u8250p(port);
+ struct mtk8250_data *data = port->private_data;
+
+ /* disable DMA for console */
+ if (uart_console(port))
+ up->dma = NULL;
+
+ if (up->dma) {
+ data->rx_status = DMA_RX_START;
+ uart_circ_clear(&port->state->xmit);
+ }
+#endif
+ memset(&port->icount, 0, sizeof(port->icount));
+
+ return serial8250_do_startup(port);
+}
+
+static void mtk8250_shutdown(struct uart_port *port)
+{
+#ifdef CONFIG_SERIAL_8250_DMA
+ struct uart_8250_port *up = up_to_u8250p(port);
+ struct mtk8250_data *data = port->private_data;
+
+ if (up->dma)
+ data->rx_status = DMA_RX_SHUTDOWN;
+#endif
+
+ return serial8250_do_shutdown(port);
+}
+
static void
mtk8250_set_termios(struct uart_port *port, struct ktermios *termios,
struct ktermios *old)
@@ -36,6 +201,17 @@ struct mtk8250_data {
unsigned long flags;
unsigned int baud, quot;
+#ifdef CONFIG_SERIAL_8250_DMA
+ if (up->dma) {
+ if (uart_console(port)) {
+ devm_kfree(up->port.dev, up->dma);
+ up->dma = NULL;
+ } else {
+ mtk8250_dma_enable(up);
+ }
+ }
+#endif
+
serial8250_do_set_termios(port, termios, old);
/*
@@ -143,9 +319,20 @@ static int __maybe_unused mtk8250_runtime_resume(struct device *dev)
pm_runtime_put_sync_suspend(port->dev);
}
+#ifdef CONFIG_SERIAL_8250_DMA
+static bool mtk8250_dma_filter(struct dma_chan *chan, void *param)
+{
+ return false;
+}
+#endif
+
static int mtk8250_probe_of(struct platform_device *pdev, struct uart_port *p,
struct mtk8250_data *data)
{
+#ifdef CONFIG_SERIAL_8250_DMA
+ int dmacnt;
+#endif
+
data->uart_clk = devm_clk_get(&pdev->dev, "baud");
if (IS_ERR(data->uart_clk)) {
/*
@@ -162,7 +349,23 @@ static int mtk8250_probe_of(struct platform_device *pdev, struct uart_port *p,
}
data->bus_clk = devm_clk_get(&pdev->dev, "bus");
- return PTR_ERR_OR_ZERO(data->bus_clk);
+ if (IS_ERR(data->bus_clk))
+ return PTR_ERR(data->bus_clk);
+
+ data->dma = NULL;
+#ifdef CONFIG_SERIAL_8250_DMA
+ dmacnt = of_property_count_strings(pdev->dev.of_node, "dma-names");
+ if (dmacnt == 2) {
+ data->dma = devm_kzalloc(&pdev->dev, sizeof(*data->dma),
+ GFP_KERNEL);
+ data->dma->fn = mtk8250_dma_filter;
+ data->dma->rx_size = MTK_UART_RX_SIZE;
+ data->dma->rxconf.src_maxburst = MTK_UART_RX_TRIGGER;
+ data->dma->txconf.dst_maxburst = MTK_UART_TX_TRIGGER;
+ }
+#endif
+
+ return 0;
}
static int mtk8250_probe(struct platform_device *pdev)
@@ -204,8 +407,14 @@ static int mtk8250_probe(struct platform_device *pdev)
uart.port.iotype = UPIO_MEM32;
uart.port.regshift = 2;
uart.port.private_data = data;
+ uart.port.shutdown = mtk8250_shutdown;
+ uart.port.startup = mtk8250_startup;
uart.port.set_termios = mtk8250_set_termios;
uart.port.uartclk = clk_get_rate(data->uart_clk);
+#ifdef CONFIG_SERIAL_8250_DMA
+ if (data->dma)
+ uart.dma = data->dma;
+#endif
/* Disable Rate Fix function */
writel(0x0, uart.port.membase +
^ permalink raw reply related
* [4/4] arm: dts: mt2701: add uart APDMA to device tree
From: Long Cheng @ 2018-09-20 6:41 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland
Cc: Matthias Brugger, Dan Williams, Greg Kroah-Hartman, Jiri Slaby,
Ed Blake, Long Cheng, dmaengine, devicetree, linux-arm-kernel,
linux-mediatek, linux-kernel, linux-serial, srv_heupstream,
Yingjoe Chen, YT Shen
1. add uart APDMA controller device node
2. add uart 0/1/2/3 DMA function
3. uart0 is console, So disable DMA
4. enable uart2 port to test DMA function.
Signed-off-by: Long Cheng <long.cheng@mediatek.com>
---
arch/arm64/boot/dts/mediatek/mt2712e.dtsi | 50 +++++++++++++++++++++++++++++
1 file changed, 50 insertions(+)
diff --git a/arch/arm64/boot/dts/mediatek/mt2712e.dtsi b/arch/arm64/boot/dts/mediatek/mt2712e.dtsi
index 593ddc7..ff94437 100644
--- a/arch/arm64/boot/dts/mediatek/mt2712e.dtsi
+++ b/arch/arm64/boot/dts/mediatek/mt2712e.dtsi
@@ -299,6 +299,9 @@
interrupts = <GIC_SPI 127 IRQ_TYPE_LEVEL_LOW>;
clocks = <&baud_clk>, <&sys_clk>;
clock-names = "baud", "bus";
+ dmas = <&apdma 10
+ &apdma 11>;
+ dma-names = "tx", "rx";
status = "disabled";
};
@@ -366,6 +369,38 @@
status = "disabled";
};
+ apdma: dma-controller@11000400 {
+ compatible = "mediatek,mt2712-uart-dma",
+ "mediatek,mt6577-uart-dma";
+ reg = <0 0x11000400 0 0x80>,
+ <0 0x11000480 0 0x80>,
+ <0 0x11000500 0 0x80>,
+ <0 0x11000580 0 0x80>,
+ <0 0x11000600 0 0x80>,
+ <0 0x11000680 0 0x80>,
+ <0 0x11000700 0 0x80>,
+ <0 0x11000780 0 0x80>,
+ <0 0x11000800 0 0x80>,
+ <0 0x11000880 0 0x80>,
+ <0 0x11000900 0 0x80>,
+ <0 0x11000980 0 0x80>;
+ interrupts = <GIC_SPI 103 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 104 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 105 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 106 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 107 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 108 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 109 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 110 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 111 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 112 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 113 IRQ_TYPE_LEVEL_LOW>,
+ <GIC_SPI 114 IRQ_TYPE_LEVEL_LOW>;
+ clocks = <&pericfg CLK_PERI_AP_DMA>;
+ clock-names = "apdma";
+ #dma-cells = <1>;
+ };
+
uart0: serial@11002000 {
compatible = "mediatek,mt2712-uart",
"mediatek,mt6577-uart";
@@ -373,6 +408,9 @@
interrupts = <GIC_SPI 91 IRQ_TYPE_LEVEL_LOW>;
clocks = <&baud_clk>, <&sys_clk>;
clock-names = "baud", "bus";
+ dmas = <&apdma 0
+ &apdma 1>;
+ dma-names = "tx", "rx";
status = "disabled";
};
@@ -383,6 +421,9 @@
interrupts = <GIC_SPI 92 IRQ_TYPE_LEVEL_LOW>;
clocks = <&baud_clk>, <&sys_clk>;
clock-names = "baud", "bus";
+ dmas = <&apdma 2
+ &apdma 3>;
+ dma-names = "tx", "rx";
status = "disabled";
};
@@ -393,6 +434,9 @@
interrupts = <GIC_SPI 93 IRQ_TYPE_LEVEL_LOW>;
clocks = <&baud_clk>, <&sys_clk>;
clock-names = "baud", "bus";
+ dmas = <&apdma 4
+ &apdma 5>;
+ dma-names = "tx", "rx";
status = "disabled";
};
@@ -403,6 +447,9 @@
interrupts = <GIC_SPI 94 IRQ_TYPE_LEVEL_LOW>;
clocks = <&baud_clk>, <&sys_clk>;
clock-names = "baud", "bus";
+ dmas = <&apdma 6
+ &apdma 7>;
+ dma-names = "tx", "rx";
status = "disabled";
};
@@ -503,6 +550,9 @@
interrupts = <GIC_SPI 126 IRQ_TYPE_LEVEL_LOW>;
clocks = <&baud_clk>, <&sys_clk>;
clock-names = "baud", "bus";
+ dmas = <&apdma 8
+ &apdma 9>;
+ dma-names = "tx", "rx";
status = "disabled";
};
^ permalink raw reply related
* [05/12] dmaengine: dw: remove dma_slave_config direction usage
From: Andy Shevchenko @ 2018-09-21 17:03 UTC (permalink / raw)
To: Vinod Koul; +Cc: dmaengine, Viresh Kumar
On Tue, Sep 11, 2018 at 02:05:29PM +0530, Vinod Koul wrote:
> dma_slave_config direction was marked as deprecated quite some
> time back, remove the usage from this driver so that the field
> can be removed
It looks like we assign direction field in custom struct any time when we call
one of ->*prep*() functions, so, it's safe to be removed from ->config().
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
>
> Signed-off-by: Vinod Koul <vkoul@kernel.org>
> ---
> CC: Viresh Kumar <vireshk@kernel.org>
> CC: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
>
> drivers/dma/dw/core.c | 5 -----
> 1 file changed, 5 deletions(-)
>
> diff --git a/drivers/dma/dw/core.c b/drivers/dma/dw/core.c
> index f43e6dafe446..d0c3e50b39fb 100644
> --- a/drivers/dma/dw/core.c
> +++ b/drivers/dma/dw/core.c
> @@ -886,12 +886,7 @@ static int dwc_config(struct dma_chan *chan, struct dma_slave_config *sconfig)
> */
> u32 s = dw->pdata->is_idma32 ? 1 : 2;
>
> - /* Check if chan will be configured for slave transfers */
> - if (!is_slave_direction(sconfig->direction))
> - return -EINVAL;
> -
> memcpy(&dwc->dma_sconfig, sconfig, sizeof(*sconfig));
> - dwc->direction = sconfig->direction;
>
> sc->src_maxburst = sc->src_maxburst > 1 ? fls(sc->src_maxburst) - s : 0;
> sc->dst_maxburst = sc->dst_maxburst > 1 ? fls(sc->dst_maxburst) - s : 0;
> --
> 2.14.4
>
^ permalink raw reply
* [07/12] dmaengine: hsu: remove dma_slave_config direction usage
From: Andy Shevchenko @ 2018-09-21 17:03 UTC (permalink / raw)
To: Vinod Koul; +Cc: dmaengine
On Tue, Sep 11, 2018 at 02:05:31PM +0530, Vinod Koul wrote:
> dma_slave_config direction was marked as deprecated quite some
> time back, remove the usage from this driver so that the field
> can be removed
>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> Signed-off-by: Vinod Koul <vkoul@kernel.org>
> ---
> CC: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
>
> drivers/dma/hsu/hsu.c | 4 ----
> 1 file changed, 4 deletions(-)
>
> diff --git a/drivers/dma/hsu/hsu.c b/drivers/dma/hsu/hsu.c
> index 202ffa9f7611..e06f20272fd7 100644
> --- a/drivers/dma/hsu/hsu.c
> +++ b/drivers/dma/hsu/hsu.c
> @@ -348,10 +348,6 @@ static int hsu_dma_slave_config(struct dma_chan *chan,
> {
> struct hsu_dma_chan *hsuc = to_hsu_dma_chan(chan);
>
> - /* Check if chan will be configured for slave transfers */
> - if (!is_slave_direction(config->direction))
> - return -EINVAL;
> -
> memcpy(&hsuc->config, config, sizeof(hsuc->config));
>
> return 0;
> --
> 2.14.4
>
^ permalink raw reply
* [08/12] dmaengine: idma: remove dma_slave_config direction usage
From: Andy Shevchenko @ 2018-09-21 17:04 UTC (permalink / raw)
To: Vinod Koul; +Cc: dmaengine
On Tue, Sep 11, 2018 at 02:05:32PM +0530, Vinod Koul wrote:
> dma_slave_config direction was marked as deprecated quite some
> time back, remove the usage from this driver so that the field
> can be removed
>
Acked-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
> Signed-off-by: Vinod Koul <vkoul@kernel.org>
> ---
> CC: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
>
> drivers/dma/idma64.c | 4 ----
> 1 file changed, 4 deletions(-)
>
> diff --git a/drivers/dma/idma64.c b/drivers/dma/idma64.c
> index 1fbf9cb9b742..40dc5d2ddf3d 100644
> --- a/drivers/dma/idma64.c
> +++ b/drivers/dma/idma64.c
> @@ -408,10 +408,6 @@ static int idma64_slave_config(struct dma_chan *chan,
> {
> struct idma64_chan *idma64c = to_idma64_chan(chan);
>
> - /* Check if chan will be configured for slave transfers */
> - if (!is_slave_direction(config->direction))
> - return -EINVAL;
> -
> memcpy(&idma64c->config, config, sizeof(idma64c->config));
>
> convert_burst(&idma64c->config.src_maxburst);
> --
> 2.14.4
>
^ permalink raw reply
* dt-bindings: rcar-dmac: Document r8a7744 support
From: Geert Uytterhoeven @ 2018-09-24 12:01 UTC (permalink / raw)
To: Biju Das
Cc: Vinod, Rob Herring, Mark Rutland, dmaengine,
open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS,
Simon Horman, Geert Uytterhoeven, Chris Paterson, Fabrizio Castro,
Linux-Renesas
On Mon, Sep 17, 2018 at 5:25 PM Biju Das <biju.das@bp.renesas.com> wrote:
> Renesas RZ/G SoC also have the R-Car gen2/3 compatible DMA controllers.
> Document RZ/G1N (also known as R8A7744) SoC bindings.
>
> Signed-off-by: Biju Das <biju.das@bp.renesas.com>
> Reviewed-by: Fabrizio Castro <fabrizio.castro@bp.renesas.com>
Reviewed-by: Geert Uytterhoeven <geert+renesas@glider.be>
Gr{oetje,eeting}s,
Geert
^ permalink raw reply
* [RFC,1/5] dmaengine: Add support for reporting DMA cached data amount
From: Peter Ujfalusi @ 2018-09-24 13:00 UTC (permalink / raw)
To: vkoul, dan.j.williams; +Cc: dmaengine, grygorii.strashko
A DMA hardware can have big cache or FIFO and the amount of data sitting in
the DMA fabric can be an interest for the clients.
For example in audio we want to know the delay in the data flow and in case
the DMA have significantly large FIFO/cache, it can affect the latenc/delay
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
---
drivers/dma/dmaengine.h | 7 +++++++
include/linux/dmaengine.h | 2 ++
2 files changed, 9 insertions(+)
diff --git a/drivers/dma/dmaengine.h b/drivers/dma/dmaengine.h
index 501c0b063f85..80214270b70e 100644
--- a/drivers/dma/dmaengine.h
+++ b/drivers/dma/dmaengine.h
@@ -77,6 +77,7 @@ static inline enum dma_status dma_cookie_status(struct dma_chan *chan,
state->last = complete;
state->used = used;
state->residue = 0;
+ state->cached = 0;
}
return dma_async_is_complete(cookie, complete, used);
}
@@ -87,6 +88,12 @@ static inline void dma_set_residue(struct dma_tx_state *state, u32 residue)
state->residue = residue;
}
+static inline void dma_set_cached(struct dma_tx_state *state, u32 cached)
+{
+ if (state)
+ state->cached = cached;
+}
+
struct dmaengine_desc_callback {
dma_async_tx_callback callback;
dma_async_tx_callback_result callback_result;
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 10ff71b13eef..16c9d021988a 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -701,11 +701,13 @@ static inline struct dma_async_tx_descriptor *txd_next(struct dma_async_tx_descr
* @residue: the remaining number of bytes left to transmit
* on the selected transfer for states DMA_IN_PROGRESS and
* DMA_PAUSED if this is implemented in the driver, else 0
+ * @cached: amount of data in bytes cached by the DMA.
*/
struct dma_tx_state {
dma_cookie_t last;
dma_cookie_t used;
u32 residue;
+ u32 cached;
};
/**
^ permalink raw reply related
* [RFC,2/5] dmaengine: Add function to request slave channel from a dma_device
From: Peter Ujfalusi @ 2018-09-24 13:00 UTC (permalink / raw)
To: vkoul, dan.j.williams; +Cc: dmaengine, grygorii.strashko
dma_get_any_slave_channel() would skip using the filter function, which
in some cases needed to be executed before the alloc_chan_resources
callback to make sure that all parameters are provided for the slave
channel.
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
---
drivers/dma/dmaengine.c | 20 ++++++++++++++++++++
include/linux/dmaengine.h | 2 ++
2 files changed, 22 insertions(+)
diff --git a/drivers/dma/dmaengine.c b/drivers/dma/dmaengine.c
index 27b6d7c2d8a0..b80f1bdd8813 100644
--- a/drivers/dma/dmaengine.c
+++ b/drivers/dma/dmaengine.c
@@ -634,6 +634,26 @@ struct dma_chan *dma_get_any_slave_channel(struct dma_device *device)
}
EXPORT_SYMBOL_GPL(dma_get_any_slave_channel);
+struct dma_chan *dmadev_get_slave_channel(struct dma_device *device,
+ dma_filter_fn fn, void *fn_param)
+{
+ dma_cap_mask_t mask;
+ struct dma_chan *chan;
+
+ dma_cap_zero(mask);
+ dma_cap_set(DMA_SLAVE, mask);
+
+ /* lock against __dma_request_channel */
+ mutex_lock(&dma_list_mutex);
+
+ chan = find_candidate(device, &mask, fn, fn_param);
+
+ mutex_unlock(&dma_list_mutex);
+
+ return IS_ERR(chan) ? NULL : chan;
+}
+EXPORT_SYMBOL_GPL(dmadev_get_slave_channel);
+
/**
* __dma_request_channel - try to allocate an exclusive channel
* @mask: capabilities that the channel must satisfy
diff --git a/include/linux/dmaengine.h b/include/linux/dmaengine.h
index 16c9d021988a..32010c6ae13a 100644
--- a/include/linux/dmaengine.h
+++ b/include/linux/dmaengine.h
@@ -1542,6 +1542,8 @@ void dma_async_device_unregister(struct dma_device *device);
void dma_run_dependencies(struct dma_async_tx_descriptor *tx);
struct dma_chan *dma_get_slave_channel(struct dma_chan *chan);
struct dma_chan *dma_get_any_slave_channel(struct dma_device *device);
+struct dma_chan *dmadev_get_slave_channel(struct dma_device *device,
+ dma_filter_fn fn, void *fn_param);
#define dma_request_channel(mask, x, y) __dma_request_channel(&(mask), x, y)
#define dma_request_slave_channel_compat(mask, x, y, dev, name) \
__dma_request_slave_channel_compat(&(mask), x, y, dev, name)
^ permalink raw reply related
* [RFC,3/5] dt-bindings: dma: ti: Add document for K3 UDMA
From: Peter Ujfalusi @ 2018-09-24 13:00 UTC (permalink / raw)
To: vkoul, dan.j.williams; +Cc: dmaengine, grygorii.strashko
New binding document for
Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P).
UDMA-P is introduced as part of the K3 architecture and can be found on
AM65x SoC.
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
---
.../devicetree/bindings/dma/ti/k3-udma.txt | 134 ++++++++++++++++++
1 file changed, 134 insertions(+)
create mode 100644 Documentation/devicetree/bindings/dma/ti/k3-udma.txt
diff --git a/Documentation/devicetree/bindings/dma/ti/k3-udma.txt b/Documentation/devicetree/bindings/dma/ti/k3-udma.txt
new file mode 100644
index 000000000000..4f79d2d30c5c
--- /dev/null
+++ b/Documentation/devicetree/bindings/dma/ti/k3-udma.txt
@@ -0,0 +1,134 @@
+* Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P)
+
+The UDMA-P is intended to perform similar (but significantly upgraded) functions
+as the packet-oriented DMA used on previous SoC devices. The UDMA-P module
+supports the transmission and reception of various packet types. The UDMA-P is
+architected to facilitate the segmentation and reassembly of SoC DMA data
+structure compliant packets to/from smaller data blocks that are natively
+compatible with the specific requirements of each connected peripheral. Multiple
+Tx and Rx channels are provided within the DMA which allow multiple segmentation
+or reassembly operations to be ongoing. The DMA controller maintains state
+information for each of the channels which allows packet segmentation and
+reassembly operations to be time division multiplexed between channels in order
+to share the underlying DMA hardware. An external DMA scheduler is used to
+control the ordering and rate at which this multiplexing occurs for Transmit
+operations. The ordering and rate of Receive operations is indirectly controlled
+by the order in which blocks are pushed into the DMA on the Rx PSI-L interface.
+
+The UDMA-P also supports acting as both a UTC and UDMA-C for its internal
+channels. Channels in the UDMA-P can be configured to be either Packet-Based or
+Third-Party channels on a channel by channel basis.
+
+Required properties:
+--------------------
+- compatible: Should be
+ "ti,am654-navss-main-udmap" for am654 main NAVSS UDMAP
+ "ti,am654-navss-mcu-udmap" for am654 mcu NAVSS UDMAP
+- #dma-cells: Should be set to <3>.
+ - The first parameter is a phandle to the remote PSI-L
+ endpoint
+ - The second parameter is the thread offset within the
+ remote thread ID range
+ - The third parameter is the channel direction.
+- reg: Memory map of UDMAP
+- reg-names: "gcfg", "rchanrt", "tchanrt"
+- interrupt-parent: phandle for "ti,sci-irq" irqchip
+- ti,ringacc: phandle for the ring accelerator node
+- ti,psil-base: PSI-L thread ID base of the UDMAP channels
+- ti,sci: phandle on TI-SCI compatible System controller node
+- ti,sci-dev-id: TI-SCI device id
+- ti,sci-rm-range-tchan: UDMA tchan resource list in pairs of type and subtype
+- ti,sci-rm-range-rchan: UDMA rchan resource list in pairs of type and subtype
+- ti,sci-rm-range-rflow: UDMA rflow resource list in pairs of type and subtype
+
+For PSI-L thread management the parent NAVSS node must have:
+- ti,sci: phandle on TI-SCI compatible System controller node
+- ti,sci-dev-id: TI-SCI device id of the NAVSS instance
+
+Remote PSI-L endpoint
+
+Required properties:
+--------------------
+- ti,psil-base: PSI-L thread ID base of the endpoint
+
+Within the PSI-L endpoint node thread configuration subnodes must present with:
+ti,psil-configX naming convention, where X is the thread ID offset.
+
+Configuration node Required properties:
+--------------------
+- linux,udma-mode: Channel mode, can be:
+ - UDMA_PKT_MODE: for Packet mode channels (peripherals)
+ - UDMA_TR_MODE: for Third-Party mode
+
+Configuration node Optional properties:
+--------------------
+- statictr-type: In case the remote endpoint requires StaticTR
+ configuration:
+ - PSIL_STATIC_TR_XY: XY type of StaticTR
+ - PSIL_STATIC_TR_MCAN: MCAN type of StaticTR
+- ti,channel-tpl: Channel Throughput level:
+ 0 / or not present - normal channel
+ 1 - High Throughput channel
+- ti,needs-epib: If the endpoint require EPIB to be present in the
+ descriptor.
+- ti,psd-size: Size of the Protocol Specific Data section of the
+ descriptor.
+
+Example:
+
+main_navss: main_navss {
+ compatible = "simple-bus";
+ #address-cells = <2>;
+ #size-cells = <2>;
+ dma-coherent;
+ dma-ranges;
+ ranges;
+
+ ti,sci = <&dmsc>;
+ ti,sci-dev-id = <118>;
+
+ main_udmap: udmap@31150000 {
+ compatible = "ti,am654-navss-main-udmap";
+ reg = <0x0 0x31150000 0x0 0x100>,
+ <0x0 0x34000000 0x0 0x100000>,
+ <0x0 0x35000000 0x0 0x100000>;
+ reg-names = "gcfg", "rchanrt", "tchanrt";
+ #dma-cells = <3>;
+
+ ti,ringacc = <&ringacc>;
+ ti,psil-base = <0x1000>;
+
+ interrupt-parent = <&main_udmass_inta>;
+
+ ti,sci = <&dmsc>;
+ ti,sci-dev-id = <188>;
+
+ ti,sci-rm-range-tchan = <0x6 0x1>, /* TX_HCHAN */
+ <0x6 0x2>; /* TX_CHAN */
+ ti,sci-rm-range-rchan = <0x6 0x4>, /* RX_HCHAN */
+ <0x6 0x5>; /* RX_CHAN */
+ ti,sci-rm-range-rflow = <0x6 0x6>; /* GP RFLOW */
+ };
+};
+
+pdma0: pdma@2a41000 {
+ compatible = "ti,am654-pdma";
+ reg = <0x0 0x02A41000 0x0 0x400>;
+ reg-names = "eccaggr_cfg";
+
+ ti,psil-base = <0x4400>;
+
+ /* ti,psil-config0-2 */
+ UDMA_PDMA_TR_XY(0);
+ UDMA_PDMA_TR_XY(1);
+ UDMA_PDMA_TR_XY(2);
+};
+
+mcasp0: mcasp@02B00000 {
+...
+ /* tx: pdma0-0, rx: pdma0-0 */
+ dmas = <&main_udmap &pdma0 0 UDMA_DIR_TX>,
+ <&main_udmap &pdma0 0 UDMA_DIR_RX>;
+ dma-names = "tx", "rx";
+...
+};
^ permalink raw reply related
* [RFC,4/5] dmaengine: ti: New driver for K3 UDMA
From: Peter Ujfalusi @ 2018-09-24 13:00 UTC (permalink / raw)
To: vkoul, dan.j.williams; +Cc: dmaengine, grygorii.strashko
DMA driver for
Texas Instruments K3 NAVSS Unified DMA – Peripheral Root Complex (UDMA-P)
The UDMA-P is intended to perform similar (but significantly upgraded) functions
as the packet-oriented DMA used on previous SoC devices. The UDMA-P module
supports the transmission and reception of various packet types. The UDMA-P is
architected to facilitate the segmentation and reassembly of SoC DMA data
structure compliant packets to/from smaller data blocks that are natively
compatible with the specific requirements of each connected peripheral. Multiple
Tx and Rx channels are provided within the DMA which allow multiple segmentation
or reassembly operations to be ongoing. The DMA controller maintains state
information for each of the channels which allows packet segmentation and
reassembly operations to be time division multiplexed between channels in order
to share the underlying DMA hardware. An external DMA scheduler is used to
control the ordering and rate at which this multiplexing occurs for Transmit
operations. The ordering and rate of Receive operations is indirectly controlled
by the order in which blocks are pushed into the DMA on the Rx PSI-L interface.
The UDMA-P also supports acting as both a UTC and UDMA-C for its internal
channels. Channels in the UDMA-P can be configured to be either Packet-Based or
Third-Party channels on a channel by channel basis.
The initial driver supports:
- MEM_TO_MEM (TR mode)
- DEV_TO_MEM (Packet / TR mode)
- MEM_TO_DEV (Packet / TR mode)
- Cyclic (Packet / TR mode)
- Metadata for descriptors
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
---
drivers/dma/ti/Kconfig | 12 +
drivers/dma/ti/Makefile | 1 +
drivers/dma/ti/k3-udma.c | 3623 +++++++++++++++++++++++++++++
drivers/dma/ti/k3-udma.h | 132 ++
include/dt-bindings/dma/k3-udma.h | 26 +
5 files changed, 3794 insertions(+)
create mode 100644 drivers/dma/ti/k3-udma.c
create mode 100644 drivers/dma/ti/k3-udma.h
create mode 100644 include/dt-bindings/dma/k3-udma.h
diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig
index e5e74e1361dc..db73371de206 100644
--- a/drivers/dma/ti/Kconfig
+++ b/drivers/dma/ti/Kconfig
@@ -33,5 +33,17 @@ config DMA_OMAP
Enable support for the TI sDMA (System DMA or DMA4) controller. This
DMA engine is found on OMAP and DRA7xx parts.
+config TI_K3_NAVSS_UDMA
+ tristate "Texas Instruments UDMA support"
+ depends on ARCH_K3 || COMPILE_TEST
+ depends on TI_SCI_PROTOCOL
+ select DMA_ENGINE
+ select DMA_VIRTUAL_CHANNELS
+ select TI_K3_NAVSS_RINGACC
+ default y
+ help
+ Enable support for the TI UDMA (Unified DMA) controller. This
+ DMA engine is used in AM65x.
+
config TI_DMA_CROSSBAR
bool
diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile
index 113e59ec9c32..911f7c723979 100644
--- a/drivers/dma/ti/Makefile
+++ b/drivers/dma/ti/Makefile
@@ -2,4 +2,5 @@
obj-$(CONFIG_TI_CPPI41) += cppi41.o
obj-$(CONFIG_TI_EDMA) += edma.o
obj-$(CONFIG_DMA_OMAP) += omap-dma.o
+obj-$(CONFIG_TI_K3_NAVSS_UDMA) += k3-udma.o
obj-$(CONFIG_TI_DMA_CROSSBAR) += dma-crossbar.o
diff --git a/drivers/dma/ti/k3-udma.c b/drivers/dma/ti/k3-udma.c
new file mode 100644
index 000000000000..2a2a2c8b1fc2
--- /dev/null
+++ b/drivers/dma/ti/k3-udma.c
@@ -0,0 +1,3623 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com
+ * Author: Peter Ujfalusi <peter.ujfalusi@ti.com>
+ */
+
+#include <linux/kernel.h>
+#include <linux/delay.h>
+#include <linux/dmaengine.h>
+#include <linux/dma-mapping.h>
+#include <linux/dmapool.h>
+#include <linux/err.h>
+#include <linux/init.h>
+#include <linux/interrupt.h>
+#include <linux/list.h>
+#include <linux/module.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/of.h>
+#include <linux/of_dma.h>
+#include <linux/of_device.h>
+#include <linux/of_irq.h>
+#include <linux/workqueue.h>
+#include <linux/completion.h>
+#include <dt-bindings/dma/k3-udma.h>
+#include <linux/soc/ti/k3-navss-ringacc.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+#include <linux/irqchip/irq-ti-sci-inta.h>
+#include <linux/soc/ti/cppi5.h>
+
+#include "../virt-dma.h"
+#include "k3-udma.h"
+
+/* element sizes */
+#define UDMA_ELSIZE_8 0
+#define UDMA_ELSIZE_16 1
+#define UDMA_ELSIZE_24 2
+#define UDMA_ELSIZE_32 3
+#define UDMA_ELSIZE_64 4
+
+static const u8 elsize_bytes[] = {
+ [UDMA_ELSIZE_8] = 1,
+ [UDMA_ELSIZE_16] = 2,
+ [UDMA_ELSIZE_24] = 3,
+ [UDMA_ELSIZE_32] = 4,
+ [UDMA_ELSIZE_64] = 8,
+};
+
+#define CPPI50_TR_FLAGS_TYPE(x) (x << 0)
+#define CPPI50_TR_FLAGS_WAIT (1 << 5)
+/*
+ * EVENT_SIZE:
+ * 0: Only generate event when the TR is completed
+ * 1: Event is generated when ICNT1 is decremented by 1
+ * 2: Event is generated when ICNT2 is decremented by 1
+ * 3: Event is generated when ICNT3 is decremented by 1
+ */
+#define CPPI50_TR_FLAGS_EVENT_SIZE(x) (((x) & 0x3) << 6)
+/*
+ * TRIGGERx:
+ * 0: No trigger
+ * 1: Global Trigger 0 for the channel
+ * 2: Global Trigger 1 for the channel
+ * 3: Local Event for the channel
+ *
+ * TRIGGER_TYPEx:
+ * 0: ICNT1 can be decremented by 1 on trigger
+ * 1: ICNT2 can be decremented by 1 on trigger
+ * 2: ICNT3 can be decremented by 1 on trigger
+ * 3: The entire TR is waiting for the trigger
+ */
+#define CPPI50_TR_FLAGS_TRIGGER0(x) (((x) & 0x3) << 8)
+#define CPPI50_TR_FLAGS_TRIGGER0_TYPE(x) (((x) & 0x3) << 10)
+#define CPPI50_TR_FLAGS_TRIGGER1(x) (((x) & 0x3) << 10)
+#define CPPI50_TR_FLAGS_TRIGGER1_TYPE(x) (((x) & 0x3) << 14)
+
+#define CPPI50_TR_FLAGS_SUPR_EVT (1 << 26)
+#define CPPI50_TR_FLAGS_EOP (1 << 31)
+
+/* Transfer Request Type 0: One Dimensional Transfer */
+struct cppi50_tr_req_type0 {
+ u32 flags;
+ u16 icnt0;
+ u16 unused;
+ u64 addr;
+} __packed;
+
+/* Transfer Request Type 1: Two Dimensional Transfer */
+struct cppi50_tr_req_type1 {
+ u32 flags;
+ u16 icnt0;
+ u16 icnt1;
+ u64 addr;
+ s32 dim1;
+ u32 padd[3]; /* full size is 32 bytes */
+} __packed;
+
+/* Transfer Request Type 2: Three Dimensional Transfer */
+struct cppi50_tr_req_type2 {
+ u32 flags;
+ u16 icnt0;
+ u16 icnt1;
+ u64 addr;
+ s32 dim1;
+ u16 icnt2;
+ u16 unused;
+ s32 dim2;
+ u32 padd; /* full size is 32 bytes */
+} __packed;
+
+/* Transfer Request Type 9: Four Dimensional Block Copy with Repacking */
+struct cppi50_tr_req_type9 {
+ u32 flags;
+ u16 icnt0;
+ u16 icnt1;
+ u64 addr;
+ s32 dim1;
+ u16 icnt2;
+ u16 icnt3;
+ s32 dim2;
+ s32 dim3;
+ u32 fmtflags;
+ s32 ddim1;
+ u64 daddr;
+ s32 ddim2;
+ s32 ddim3;
+ u16 dicnt0;
+ u16 dicnt1;
+ u16 dicnt2;
+ u16 dicnt3;
+} __packed;
+
+/* Transfer Request Type 10: Two Dimensional BLock Copy */
+struct cppi50_tr_req_type10 {
+ u32 flags;
+ u16 icnt0;
+ u16 icnt1;
+ u64 addr;
+ s32 dim1;
+ u32 unused[3];
+ u32 fmtflags;
+ s32 ddim1;
+ u64 daddr;
+ u32 padd[4]; /* full size is 64 bytes */
+} __packed;
+
+/*
+ * Transfer Request Type 15: Four Dimensional Block Copy with Repacking
+ * and Indirection Support
+ */
+struct cppi50_tr_req_type15 {
+ u32 flags;
+ u16 icnt0;
+ u16 icnt1;
+ u64 addr;
+ s32 dim1;
+ u16 icnt2;
+ u16 icnt3;
+ s32 dim2;
+ s32 dim3;
+ u32 unused;
+ s32 ddim1;
+ u64 daddr;
+ s32 ddim2;
+ s32 ddim3;
+ u16 dicnt0;
+ u16 dicnt1;
+ u16 dicnt2;
+ u16 dicnt3;
+} __packed;
+
+struct cppi50_tr_resp {
+ u8 status;
+ u8 reserved;
+ u8 cmd_id;
+ u8 flags;
+} __packed;
+
+#define CPPI50_TRDESC_W0_LAST_ENTRY(x) (((x) & 0x3fff) << 0)
+#define CPPI50_TRDESC_W0_RELOAD_IDX(x) (((x) & 0x3fff) << 14)
+#define CPPI50_TRDESC_W0_RELOAD_CNT(x) (((x) & 0x1ff) << 20)
+#define CPPI50_TRDESC_W0_TYPE (0x3 << 30)
+
+#define CPPI50_TRDESC_W1_FLOWID(x) (((x) & 0x3fff) << 0)
+#define CPPI50_TRDESC_W1_PACKETID(x) (((x) & 0x3ff) << 14)
+#define CPPI50_TRDESC_W1_TR_SIZE_16 (0 << 24)
+#define CPPI50_TRDESC_W1_TR_SIZE_32 (1 << 24)
+#define CPPI50_TRDESC_W1_TR_SIZE_64 (2 << 24)
+#define CPPI50_TRDESC_W1_TR_SIZE_128 (3 << 24)
+
+/* CPPI 5.0 Transfer Request Descriptor */
+struct cppi50_tr_req_desc {
+ u32 packet_info[4];
+} __aligned(64) __packed;
+
+struct udma_static_tr {
+ u8 elsize; /* RPSTR0 */
+ u16 elcnt; /* RPSTR0 */
+ u16 bstcnt; /* RPSTR1 */
+};
+
+#define K3_UDMA_MAX_RFLOWS 1024
+
+struct udma_chan;
+
+enum udma_mmr {
+ MMR_GCFG = 0,
+ MMR_RCHANRT,
+ MMR_TCHANRT,
+ MMR_LAST,
+};
+
+static const char * const mmr_names[] = { "gcfg", "rchanrt", "tchanrt" };
+
+struct udma_tchan {
+ void __iomem *reg_rt;
+
+ int id;
+ struct k3_nav_ring *t_ring; /* Transmit ring */
+ struct k3_nav_ring *tc_ring; /* Transmit Completion ring */
+};
+
+struct udma_rchan {
+ void __iomem *reg_rt;
+
+ int id;
+ struct k3_nav_ring *fd_ring; /* Free Descriptor ring */
+ struct k3_nav_ring *r_ring; /* Receive ring*/
+};
+
+struct udma_rflow {
+ void __iomem *reg_rflow;
+
+ int id;
+};
+
+struct udma_match_data {
+ u8 tpl_levels;
+ u32 level_start_idx[];
+};
+
+struct udma_dev {
+ struct dma_device ddev;
+ struct device *dev;
+ void __iomem *mmrs[MMR_LAST];
+ const struct udma_match_data *match_data;
+
+ struct udma_tisci_rm tisci_rm;
+
+ struct k3_nav_ringacc *ringacc;
+
+ struct irq_domain *irq_domain;
+
+ struct work_struct purge_work;
+ struct list_head desc_to_purge;
+ spinlock_t lock;
+
+ int tchan_cnt;
+ int echan_cnt;
+ int rchan_cnt;
+ int rflow_cnt;
+ unsigned long *tchan_map;
+ unsigned long *rchan_map;
+ unsigned long *rflow_map;
+ unsigned long *rflow_map_reserved;
+
+ struct udma_tchan *tchans;
+ struct udma_rchan *rchans;
+ struct udma_rflow *rflows;
+
+ struct udma_chan *channels;
+ u32 psil_base;
+};
+
+/*
+ * Slave RX scatter gather workaround:
+ * We need to use single continuous buffer if the original buffer is scattered
+ */
+struct udma_rx_sg_workaround {
+ bool in_use;
+
+ struct scatterlist *sgl;
+ unsigned int sglen;
+ size_t total_len;
+
+ struct scatterlist single_sg;
+};
+
+struct udma_hwdesc {
+ size_t cppi5_desc_size;
+ void *cppi5_desc_vaddr;
+ dma_addr_t cppi5_desc_paddr;
+
+ /* TR descriptor internal pointers */
+ void *tr_req_base;
+ struct cppi50_tr_resp *tr_resp_base;
+};
+
+struct udma_desc {
+ struct virt_dma_desc vd;
+
+ bool terminated;
+
+ enum dma_transfer_direction dir;
+
+ struct udma_static_tr static_tr;
+ u32 residue;
+
+ unsigned int sglen;
+ unsigned int desc_idx; /* Only used for cyclic in packet mode */
+ unsigned int tr_idx;
+
+ /* for slave_sg RX workaround */
+ struct udma_rx_sg_workaround rx_sg_wa;
+
+ u32 metadata_size;
+ void *metadata; /* pointer to provided metadata buffer (EPIP, PSdata) */
+
+ unsigned int hwdesc_count;
+ struct udma_hwdesc hwdesc[0];
+};
+
+enum udma_chan_state {
+ UDMA_CHAN_IS_IDLE = 0, /* not active, no teardown is in progress */
+ UDMA_CHAN_IS_ACTIVE, /* Normal operation */
+ UDMA_CHAN_IS_ACTIVE_FLUSH, /* Flushing for delayed tx */
+ UDMA_CHAN_IS_TERMINATING, /* channel is being terminated */
+};
+
+struct udma_chan {
+ struct virt_dma_chan vc;
+ struct dma_slave_config cfg;
+ struct udma_dev *ud;
+ struct udma_desc *desc;
+ struct udma_desc *terminated_desc;
+ struct udma_static_tr static_tr;
+ char *name;
+
+ struct udma_tchan *tchan;
+ struct udma_rchan *rchan;
+ struct udma_rflow *rflow;
+
+ bool psil_paired;
+ u32 irq_ra_tisci;
+ u32 irq_ra_idx;
+ u32 irq_udma_idx;
+
+ int irq_num_ring;
+ int irq_num_udma;
+
+ bool cyclic;
+ bool paused;
+
+ enum udma_chan_state state;
+ struct completion teardown_completed;
+
+ u32 bcnt; /* number of bytes completed since the start of the channel */
+ u32 in_ring_cnt; /* number of descriptors in flight */
+
+ bool pkt_mode; /* TR or packet */
+ bool needs_epib; /* EPIB is needed for the communication or not */
+ u32 psd_size; /* size of Protocol Specific Data */
+ u32 metadata_size; /* (needs_epib ? 16:0) + psd_size */
+ u32 hdesc_size; /* Size of a packet descriptor in packet mode */
+ int desc_align; /* alignment to use for descriptors */
+ int slave_thread_id;
+ u32 src_thread;
+ u32 dst_thread;
+ u32 static_tr_type;
+ enum udma_tp_level channel_tpl; /* Channel Throughput Level */
+
+ /* dmapool for packet mode descriptors */
+ bool use_dma_pool;
+ struct dma_pool *hdesc_pool;
+
+ u32 id;
+ enum dma_transfer_direction dir;
+};
+
+static inline struct udma_dev *to_udma_dev(struct dma_device *d)
+{
+ return container_of(d, struct udma_dev, ddev);
+}
+
+static inline struct udma_chan *to_udma_chan(struct dma_chan *c)
+{
+ return container_of(c, struct udma_chan, vc.chan);
+}
+
+static inline struct udma_desc *to_udma_desc(struct dma_async_tx_descriptor *t)
+{
+ return container_of(t, struct udma_desc, vd.tx);
+}
+
+#define UDMA_CH_1000(ch) (ch * 0x1000)
+
+/* Generic register access functions */
+static inline u32 udma_read(void __iomem *base, int reg)
+{
+ return __raw_readl(base + reg);
+}
+
+static inline void udma_write(void __iomem *base, int reg, u32 val)
+{
+ __raw_writel(val, base + reg);
+}
+
+static inline void udma_update_bits(void __iomem *base, int reg,
+ u32 mask, u32 val)
+{
+ u32 tmp, orig;
+
+ orig = __raw_readl(base + reg);
+ tmp = orig & ~mask;
+ tmp |= (val & mask);
+
+ if (tmp != orig)
+ __raw_writel(tmp, base + reg);
+}
+
+/* TCHANRT */
+static inline u32 udma_tchanrt_read(struct udma_tchan *tchan, int reg)
+{
+ if (!tchan)
+ return 0;
+ return udma_read(tchan->reg_rt, reg);
+}
+
+static inline void udma_tchanrt_write(struct udma_tchan *tchan, int reg,
+ u32 val)
+{
+ if (!tchan)
+ return;
+ udma_write(tchan->reg_rt, reg, val);
+}
+
+static inline void udma_tchanrt_update_bits(struct udma_tchan *tchan, int reg,
+ u32 mask, u32 val)
+{
+ if (!tchan)
+ return;
+ udma_update_bits(tchan->reg_rt, reg, mask, val);
+}
+
+/* RCHANRT */
+static inline u32 udma_rchanrt_read(struct udma_rchan *rchan, int reg)
+{
+ if (!rchan)
+ return 0;
+ return udma_read(rchan->reg_rt, reg);
+}
+
+static inline void udma_rchanrt_write(struct udma_rchan *rchan, int reg,
+ u32 val)
+{
+ if (!rchan)
+ return;
+ udma_write(rchan->reg_rt, reg, val);
+}
+
+static inline void udma_rchanrt_update_bits(struct udma_rchan *rchan, int reg,
+ u32 mask, u32 val)
+{
+ if (!rchan)
+ return;
+ udma_update_bits(rchan->reg_rt, reg, mask, val);
+}
+
+static inline int navss_psil_pair(struct udma_dev *ud, u32 src_thread,
+ u32 dst_thread)
+{
+ struct udma_tisci_rm *tisci_rm = &ud->tisci_rm;
+
+ dst_thread |= UDMA_PSIL_DST_THREAD_ID_OFFSET;
+ return tisci_rm->tisci_psil_ops->pair(tisci_rm->tisci,
+ tisci_rm->tisci_navss_dev_id,
+ src_thread, dst_thread);
+}
+
+static inline int navss_psil_unpair(struct udma_dev *ud, u32 src_thread,
+ u32 dst_thread)
+{
+ struct udma_tisci_rm *tisci_rm = &ud->tisci_rm;
+
+ dst_thread |= UDMA_PSIL_DST_THREAD_ID_OFFSET;
+ return tisci_rm->tisci_psil_ops->unpair(tisci_rm->tisci,
+ tisci_rm->tisci_navss_dev_id,
+ src_thread, dst_thread);
+}
+
+static inline char *udma_get_dir_text(enum dma_transfer_direction dir)
+{
+ switch (dir) {
+ case DMA_DEV_TO_MEM:
+ return "DEV_TO_MEM";
+ case DMA_MEM_TO_DEV:
+ return "MEM_TO_DEV";
+ case DMA_MEM_TO_MEM:
+ return "MEM_TO_MEM";
+ case DMA_DEV_TO_DEV:
+ return "DEV_TO_DEV";
+ default:
+ break;
+ }
+
+ return "invalid";
+}
+
+static inline void udma_dump_chan_stdata(struct udma_chan *uc)
+{
+ struct device *dev = uc->ud->dev;
+ u32 offset;
+ int i;
+
+ if (uc->dir == DMA_MEM_TO_DEV || uc->dir == DMA_MEM_TO_MEM) {
+ dev_dbg(dev, "TCHAN State data:\n");
+ for (i = 0; i < 32; i++) {
+ offset = UDMA_TCHAN_RT_STDATA_REG + i * 4;
+ dev_dbg(dev, "TRT_STDATA[%02d]: 0x%08x\n", i,
+ udma_tchanrt_read(uc->tchan, offset));
+ }
+ }
+
+ if (uc->dir == DMA_DEV_TO_MEM || uc->dir == DMA_MEM_TO_MEM) {
+ dev_dbg(dev, "RCHAN State data:\n");
+ for (i = 0; i < 32; i++) {
+ offset = UDMA_RCHAN_RT_STDATA_REG + i * 4;
+ dev_dbg(dev, "RRT_STDATA[%02d]: 0x%08x\n", i,
+ udma_rchanrt_read(uc->rchan, offset));
+ }
+ }
+}
+
+static inline dma_addr_t udma_curr_cppi5_desc_paddr(struct udma_desc *d,
+ int idx)
+{
+ return d->hwdesc[idx].cppi5_desc_paddr;
+}
+
+static inline void *udma_curr_cppi5_desc_vaddr(struct udma_desc *d, int idx)
+{
+ return d->hwdesc[idx].cppi5_desc_vaddr;
+}
+
+static inline struct udma_desc *udma_udma_desc_from_paddr(struct udma_chan *uc,
+ dma_addr_t paddr)
+{
+ struct udma_desc *d = uc->terminated_desc;
+
+ if (d) {
+ dma_addr_t desc_paddr = udma_curr_cppi5_desc_paddr(d,
+ d->desc_idx);
+
+ if (desc_paddr != paddr)
+ d = NULL;
+ }
+
+ if (!d) {
+ d = uc->desc;
+ if (d) {
+ dma_addr_t desc_paddr = udma_curr_cppi5_desc_paddr(d,
+ d->desc_idx);
+
+ if (desc_paddr != paddr)
+ d = NULL;
+ }
+ }
+
+ return d;
+}
+
+static inline size_t udma_calc_trdesc_size(size_t tr_size, int tr_count)
+{
+ size_t trdesc_size = sizeof(struct cppi50_tr_req_desc);
+
+ trdesc_size += tr_size * (tr_count + 1);
+ trdesc_size += tr_count * sizeof(struct cppi50_tr_resp);
+
+ return trdesc_size;
+}
+
+static void udma_free_hwdesc(struct virt_dma_desc *vd)
+{
+ struct udma_chan *uc = to_udma_chan(vd->tx.chan);
+ struct udma_desc *d = to_udma_desc(&vd->tx);
+
+ if (uc->use_dma_pool) {
+ int i;
+
+ for (i = 0; i < d->hwdesc_count; i++) {
+ if (!d->hwdesc[i].cppi5_desc_vaddr)
+ continue;
+
+ dma_pool_free(uc->hdesc_pool,
+ d->hwdesc[i].cppi5_desc_vaddr,
+ d->hwdesc[i].cppi5_desc_paddr);
+
+ d->hwdesc[i].cppi5_desc_vaddr = NULL;
+ }
+ } else if (d->hwdesc[0].cppi5_desc_vaddr) {
+ struct udma_dev *ud = to_udma_dev(vd->tx.chan->device);
+
+ dma_free_coherent(ud->dev, d->hwdesc[0].cppi5_desc_size,
+ d->hwdesc[0].cppi5_desc_vaddr,
+ d->hwdesc[0].cppi5_desc_paddr);
+
+ d->hwdesc[0].cppi5_desc_vaddr = NULL;
+ }
+
+ if (d->rx_sg_wa.in_use) {
+ dma_unmap_sg(uc->ud->dev, &d->rx_sg_wa.single_sg, 1,
+ DMA_FROM_DEVICE);
+ kfree(sg_virt(&d->rx_sg_wa.single_sg));
+ }
+}
+
+static void udma_purge_desc_work(struct work_struct *work)
+{
+ struct udma_dev *ud = container_of(work, typeof(*ud), purge_work);
+ struct virt_dma_desc *vd, *_vd;
+ unsigned long flags;
+ LIST_HEAD(head);
+
+ spin_lock_irqsave(&ud->lock, flags);
+ list_splice_tail_init(&ud->desc_to_purge, &head);
+ spin_unlock_irqrestore(&ud->lock, flags);
+
+ list_for_each_entry_safe(vd, _vd, &head, node) {
+ struct udma_desc *d;
+
+ d = to_udma_desc(&vd->tx);
+
+ udma_free_hwdesc(vd);
+ list_del(&vd->node);
+ kfree(d);
+ }
+
+ /* If more to purge, schedule the work again */
+ if (!list_empty(&ud->desc_to_purge))
+ schedule_work(&ud->purge_work);
+}
+
+static void udma_desc_free(struct virt_dma_desc *vd)
+{
+ struct udma_dev *ud = to_udma_dev(vd->tx.chan->device);
+ struct udma_chan *uc = to_udma_chan(vd->tx.chan);
+ struct udma_desc *d = to_udma_desc(&vd->tx);
+ unsigned long flags;
+
+ if (uc->terminated_desc == d)
+ uc->terminated_desc = NULL;
+
+ if (uc->use_dma_pool) {
+ udma_free_hwdesc(&d->vd);
+ kfree(d);
+ return;
+ }
+
+ spin_lock_irqsave(&ud->lock, flags);
+ list_add_tail(&vd->node, &ud->desc_to_purge);
+ spin_unlock_irqrestore(&ud->lock, flags);
+
+ schedule_work(&ud->purge_work);
+}
+
+static inline bool udma_is_chan_running(struct udma_chan *uc)
+{
+ u32 trt_ctl = 0;
+ u32 rrt_ctl = 0;
+
+ switch (uc->dir) {
+ case DMA_DEV_TO_MEM:
+ rrt_ctl = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_CTL_REG);
+ break;
+ case DMA_MEM_TO_DEV:
+ trt_ctl = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_CTL_REG);
+ break;
+ case DMA_MEM_TO_MEM:
+ trt_ctl = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_CTL_REG);
+ rrt_ctl = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_CTL_REG);
+ break;
+ default:
+ break;
+ }
+
+ if (trt_ctl & UDMA_CHAN_RT_CTL_EN || rrt_ctl & UDMA_CHAN_RT_CTL_EN)
+ return true;
+
+ return false;
+}
+
+static void udma_sync_for_device(struct udma_chan *uc, int idx)
+{
+ struct udma_desc *d = uc->desc;
+
+ if (uc->cyclic && uc->pkt_mode) {
+ dma_sync_single_for_device(uc->ud->dev,
+ d->hwdesc[idx].cppi5_desc_paddr,
+ d->hwdesc[idx].cppi5_desc_size,
+ DMA_TO_DEVICE);
+ } else {
+ int i;
+
+ for (i = 0; i < d->hwdesc_count; i++) {
+ if (!d->hwdesc[i].cppi5_desc_vaddr)
+ continue;
+
+ dma_sync_single_for_device(uc->ud->dev,
+ d->hwdesc[i].cppi5_desc_paddr,
+ d->hwdesc[i].cppi5_desc_size,
+ DMA_TO_DEVICE);
+ }
+ }
+}
+
+static int udma_push_to_ring(struct udma_chan *uc, int idx)
+{
+ struct udma_desc *d = uc->desc;
+
+ struct k3_nav_ring *ring = NULL;
+ int ret = -EINVAL;
+
+ switch (uc->dir) {
+ case DMA_DEV_TO_MEM:
+ ring = uc->rchan->fd_ring;
+ break;
+ case DMA_MEM_TO_DEV:
+ ring = uc->tchan->t_ring;
+ break;
+ case DMA_MEM_TO_MEM:
+ ring = uc->tchan->t_ring;
+ break;
+ default:
+ break;
+ }
+
+ if (ring) {
+ dma_addr_t desc_addr = udma_curr_cppi5_desc_paddr(d, idx);
+
+ wmb(); /* Ensure that writes are not moved over this point */
+ udma_sync_for_device(uc, idx);
+ ret = k3_nav_ringacc_ring_push(ring, &desc_addr);
+ uc->in_ring_cnt++;
+ }
+
+ return ret;
+}
+
+static int udma_pop_from_ring(struct udma_chan *uc, dma_addr_t *addr)
+{
+ struct k3_nav_ring *ring = NULL;
+ int ret = -ENOENT;
+
+ switch (uc->dir) {
+ case DMA_DEV_TO_MEM:
+ ring = uc->rchan->r_ring;
+ break;
+ case DMA_MEM_TO_DEV:
+ ring = uc->tchan->tc_ring;
+ break;
+ case DMA_MEM_TO_MEM:
+ ring = uc->tchan->tc_ring;
+ break;
+ default:
+ break;
+ }
+
+ if (ring && k3_nav_ringacc_ring_get_occ(ring)) {
+ struct udma_desc *d = NULL;
+
+ ret = k3_nav_ringacc_ring_pop(ring, addr);
+ if (ret)
+ return ret;
+
+ /* Teardown completion */
+ if (*addr & 0x1)
+ return ret;
+
+ d = udma_udma_desc_from_paddr(uc, *addr);
+
+ if (d)
+ dma_sync_single_for_cpu(uc->ud->dev, *addr,
+ d->hwdesc[0].cppi5_desc_size,
+ DMA_FROM_DEVICE);
+ rmb(); /* Ensure that reads are not moved before this point */
+
+ if (!ret)
+ uc->in_ring_cnt--;
+ }
+
+ return ret;
+}
+
+static void udma_reset_rings(struct udma_chan *uc)
+{
+ struct k3_nav_ring *ring1 = NULL;
+ struct k3_nav_ring *ring2 = NULL;
+
+ switch (uc->dir) {
+ case DMA_DEV_TO_MEM:
+ if (uc->rchan) {
+ ring1 = uc->rchan->fd_ring;
+ ring2 = uc->rchan->r_ring;
+ }
+ break;
+ case DMA_MEM_TO_DEV:
+ if (uc->tchan) {
+ ring1 = uc->tchan->t_ring;
+ ring2 = uc->tchan->tc_ring;
+ }
+ break;
+ case DMA_MEM_TO_MEM:
+ if (uc->tchan) {
+ ring1 = uc->tchan->t_ring;
+ ring2 = uc->tchan->tc_ring;
+ }
+ break;
+ default:
+ break;
+ }
+
+ if (ring1)
+ k3_nav_ringacc_ring_reset_dma(ring1, 0);
+ if (ring2)
+ k3_nav_ringacc_ring_reset(ring2);
+
+ /* make sure we are not leaking memory by stalled descriptor */
+ if (uc->terminated_desc) {
+ udma_desc_free(&uc->terminated_desc->vd);
+ uc->terminated_desc = NULL;
+ }
+
+ uc->in_ring_cnt = 0;
+}
+
+static inline void udma_reset_counters(struct udma_chan *uc)
+{
+ u32 val;
+
+ if (uc->tchan) {
+ val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_BCNT_REG);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_BCNT_REG, val);
+
+ val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_SBCNT_REG);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_SBCNT_REG, val);
+
+ val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_PCNT_REG);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PCNT_REG, val);
+
+ val = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_PEER_BCNT_REG);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_BCNT_REG, val);
+ }
+
+ if (uc->rchan) {
+ val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_BCNT_REG);
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_BCNT_REG, val);
+
+ val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_SBCNT_REG);
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_SBCNT_REG, val);
+
+ val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_PCNT_REG);
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PCNT_REG, val);
+
+ val = udma_rchanrt_read(uc->rchan, UDMA_RCHAN_RT_PEER_BCNT_REG);
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_BCNT_REG, val);
+ }
+
+ uc->bcnt = 0;
+}
+
+static inline int udma_reset_chan(struct udma_chan *uc, bool hard)
+{
+ switch (uc->dir) {
+ case DMA_DEV_TO_MEM:
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_RT_EN_REG, 0);
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG, 0);
+ break;
+ case DMA_MEM_TO_DEV:
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, 0);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_RT_EN_REG, 0);
+ break;
+ case DMA_MEM_TO_MEM:
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG, 0);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG, 0);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ /* Reset all counters */
+ udma_reset_counters(uc);
+
+ /* Hard reset: re-initialize the channel to reset */
+ if (hard) {
+ int ret;
+
+ uc->ud->ddev.device_free_chan_resources(&uc->vc.chan);
+ ret = uc->ud->ddev.device_alloc_chan_resources(&uc->vc.chan);
+ if (ret)
+ return ret;
+ }
+ uc->state = UDMA_CHAN_IS_IDLE;
+
+ return 0;
+}
+
+static inline void udma_start_desc(struct udma_chan *uc)
+{
+ if (uc->cyclic && uc->pkt_mode) {
+ int i;
+
+ /* Push all descriptors to ring for cyclic packet mode */
+ for (i = 0; i < uc->desc->sglen; i++)
+ udma_push_to_ring(uc, i);
+ } else {
+ udma_push_to_ring(uc, 0);
+ }
+}
+
+static inline bool udma_chan_needs_reconfiguration(struct udma_chan *uc)
+{
+ /* Only PDMAs have staticTR */
+ if (!uc->static_tr_type)
+ return false;
+
+ /* RX channels always need to be reset, reconfigured */
+ if (uc->dir == DMA_DEV_TO_MEM)
+ return true;
+
+ /* Check if the staticTR configuration has changed for TX */
+ if (memcmp(&uc->static_tr, &uc->desc->static_tr, sizeof(uc->static_tr)))
+ return true;
+
+ return false;
+}
+
+static int udma_start(struct udma_chan *uc)
+{
+ struct virt_dma_desc *vd = vchan_next_desc(&uc->vc);
+
+ if (!vd) {
+ uc->desc = NULL;
+ return -ENOENT;
+ }
+
+ list_del(&vd->node);
+
+ uc->desc = to_udma_desc(&vd->tx);
+
+ /* Channel is already running and does not need reconfiguration */
+ if (udma_is_chan_running(uc) && !udma_chan_needs_reconfiguration(uc)) {
+ udma_start_desc(uc);
+ goto out;
+ }
+
+ /* Make sure that we clear the teardown bit, if it is set */
+ udma_reset_chan(uc, false);
+
+ /* Push descriptors before we start the channel */
+ udma_start_desc(uc);
+
+ switch (uc->desc->dir) {
+ case DMA_DEV_TO_MEM:
+ /* Config remote TR */
+ if (uc->static_tr_type) {
+ udma_rchanrt_write(uc->rchan,
+ UDMA_RCHAN_RT_PEER_STATIC_TR_XY_REG,
+ PDMA_STATIC_TR_Y(uc->desc->static_tr.elcnt) |
+ PDMA_STATIC_TR_X(uc->desc->static_tr.elsize));
+ udma_rchanrt_write(uc->rchan,
+ UDMA_RCHAN_RT_PEER_STATIC_TR_Z_REG,
+ PDMA_STATIC_TR_Z(uc->desc->static_tr.bstcnt));
+
+ /* save the current staticTR configuration */
+ memcpy(&uc->static_tr, &uc->desc->static_tr,
+ sizeof(uc->static_tr));
+ }
+
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN);
+
+ /* Enable remote */
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_ENABLE);
+
+ break;
+ case DMA_MEM_TO_DEV:
+ /* Config remote TR */
+ if (uc->static_tr_type) {
+ udma_tchanrt_write(uc->tchan,
+ UDMA_TCHAN_RT_PEER_STATIC_TR_XY_REG,
+ PDMA_STATIC_TR_Y(uc->desc->static_tr.elcnt) |
+ PDMA_STATIC_TR_X(uc->desc->static_tr.elsize));
+
+ /* save the current staticTR configuration */
+ memcpy(&uc->static_tr, &uc->desc->static_tr,
+ sizeof(uc->static_tr));
+ }
+
+ /* Enable remote */
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_ENABLE);
+
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN);
+
+ break;
+ case DMA_MEM_TO_MEM:
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN);
+
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ uc->state = UDMA_CHAN_IS_ACTIVE;
+out:
+
+ return 0;
+}
+
+static inline int udma_stop(struct udma_chan *uc)
+{
+ enum udma_chan_state old_state = uc->state;
+
+ uc->state = UDMA_CHAN_IS_TERMINATING;
+ reinit_completion(&uc->teardown_completed);
+
+ switch (uc->dir) {
+ case DMA_DEV_TO_MEM:
+ udma_rchanrt_write(uc->rchan, UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_ENABLE |
+ UDMA_PEER_RT_EN_TEARDOWN);
+ break;
+ case DMA_MEM_TO_DEV:
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_ENABLE |
+ UDMA_PEER_RT_EN_FLUSH);
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN |
+ UDMA_CHAN_RT_CTL_TDOWN);
+ break;
+ case DMA_MEM_TO_MEM:
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN |
+ UDMA_CHAN_RT_CTL_TDOWN);
+ break;
+ default:
+ uc->state = old_state;
+ complete_all(&uc->teardown_completed);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static void udma_cyclic_packet_elapsed(struct udma_chan *uc)
+{
+ struct udma_desc *d = uc->desc;
+ struct knav_udmap_host_desc_t *h_desc;
+
+ h_desc = d->hwdesc[d->desc_idx].cppi5_desc_vaddr;
+ knav_udmap_hdesc_reset_to_original(h_desc);
+ udma_push_to_ring(uc, d->desc_idx);
+ d->desc_idx = (d->desc_idx + 1) % d->sglen;
+}
+
+static inline void udma_fetch_epib(struct udma_chan *uc, struct udma_desc *d)
+{
+ struct knav_udmap_host_desc_t *h_desc = d->hwdesc[0].cppi5_desc_vaddr;
+
+ memcpy(d->metadata, h_desc->epib, d->metadata_size);
+}
+
+static inline bool udma_is_desc_really_done(struct udma_chan *uc,
+ struct udma_desc *d)
+{
+ u32 peer_bcnt, bcnt;
+
+ /* Only TX towards PDMA is affected */
+ if (!uc->static_tr_type || uc->dir != DMA_MEM_TO_DEV)
+ return true;
+
+ peer_bcnt = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_PEER_BCNT_REG);
+ bcnt = udma_tchanrt_read(uc->tchan, UDMA_TCHAN_RT_BCNT_REG);
+
+ if (peer_bcnt < bcnt)
+ return false;
+
+ return true;
+}
+
+static void udma_flush_tx(struct udma_chan *uc)
+{
+ if (uc->dir != DMA_MEM_TO_DEV)
+ return;
+
+ uc->state = UDMA_CHAN_IS_ACTIVE_FLUSH;
+
+ udma_tchanrt_write(uc->tchan, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN |
+ UDMA_CHAN_RT_CTL_TDOWN);
+}
+
+static void udma_ring_callback(struct udma_chan *uc, dma_addr_t paddr)
+{
+ struct udma_desc *d;
+ unsigned long flags;
+
+ if (!paddr)
+ return;
+
+ spin_lock_irqsave(&uc->vc.lock, flags);
+
+ /* Teardown completion message */
+ if (paddr & 0x1) {
+ /* Compensate our internal pop/push counter */
+ uc->in_ring_cnt++;
+
+ complete_all(&uc->teardown_completed);
+
+ if (uc->terminated_desc) {
+ udma_desc_free(&uc->terminated_desc->vd);
+ uc->terminated_desc = NULL;
+ }
+
+ if (!uc->desc)
+ udma_start(uc);
+
+ if (uc->state != UDMA_CHAN_IS_ACTIVE_FLUSH)
+ goto out;
+ else if (uc->desc)
+ paddr = udma_curr_cppi5_desc_paddr(uc->desc,
+ uc->desc->desc_idx);
+ }
+
+ d = udma_udma_desc_from_paddr(uc, paddr);
+
+ if (d) {
+ dma_addr_t desc_paddr = udma_curr_cppi5_desc_paddr(d,
+ d->desc_idx);
+ if (desc_paddr != paddr) {
+ dev_err(uc->ud->dev, "not matching descriptors!\n");
+ goto out;
+ }
+
+ if (uc->cyclic) {
+ /* push the descriptor back to the ring */
+ if (d == uc->desc) {
+ udma_cyclic_packet_elapsed(uc);
+ vchan_cyclic_callback(&d->vd);
+ }
+ } else {
+ bool desc_done = true;
+
+ if (d == uc->desc) {
+ desc_done = udma_is_desc_really_done(uc, d);
+
+ if (desc_done) {
+ uc->bcnt += d->residue;
+ udma_start(uc);
+ } else {
+ udma_flush_tx(uc);
+ }
+ } else if (d == uc->terminated_desc) {
+ uc->terminated_desc = NULL;
+ }
+
+ if (desc_done)
+ vchan_cookie_complete(&d->vd);
+ }
+ }
+out:
+ spin_unlock_irqrestore(&uc->vc.lock, flags);
+}
+
+static void udma_tr_event_callback(struct udma_chan *uc)
+{
+ struct udma_desc *d;
+ unsigned long flags;
+
+ spin_lock_irqsave(&uc->vc.lock, flags);
+ d = uc->desc;
+ if (d) {
+ d->tr_idx = (d->tr_idx + 1) % d->sglen;
+
+ if (uc->cyclic) {
+ vchan_cyclic_callback(&d->vd);
+ } else {
+ /* TODO: figure out the real amount of data */
+ uc->bcnt += d->residue;
+ udma_start(uc);
+ vchan_cookie_complete(&d->vd);
+ }
+ }
+
+ spin_unlock_irqrestore(&uc->vc.lock, flags);
+}
+
+static irqreturn_t udma_ring_irq_handler(int irq, void *data)
+{
+ struct udma_chan *uc = data;
+ dma_addr_t paddr = 0;
+
+ if (!udma_pop_from_ring(uc, &paddr))
+ udma_ring_callback(uc, paddr);
+
+ return IRQ_HANDLED;
+}
+
+static irqreturn_t udma_udma_irq_handler(int irq, void *data)
+{
+ struct udma_chan *uc = data;
+ struct udma_tisci_rm *tisci_rm = &uc->ud->tisci_rm;
+
+ ti_sci_inta_ack_event(uc->ud->irq_domain, tisci_rm->tisci_dev_id,
+ uc->irq_udma_idx, uc->irq_num_udma);
+
+ udma_tr_event_callback(uc);
+
+ return IRQ_HANDLED;
+}
+
+/**
+ * __udma_reserve_rflow_range - reserve range of flow ids
+ * @ud: UDMA device
+ * @from: Start the search from this flow id number
+ * @cnt: Number of consecutive flow ids to allocate
+ *
+ * Reserve range of flow ids for future use, those flows can be allocated
+ * only using explicit flow id number. if @from is set to -1 it will try to find
+ * first free range. if @from is positive value it will force allocation only
+ * of the specified range of flows.
+ *
+ * Returns -ENOMEM if can't find free range.
+ * -EEXIST if requested range is busy.
+ * -EINVAL if wrong input values passed.
+ * Returns flow id on success.
+ */
+static int __udma_reserve_rflow_range(struct udma_dev *ud, int from, int cnt)
+{
+ int start, tmp_from;
+ DECLARE_BITMAP(tmp, K3_UDMA_MAX_RFLOWS);
+
+ tmp_from = from;
+ if (tmp_from < 0)
+ tmp_from = ud->rchan_cnt;
+ /* default flows can't be reserved and accessible only by id */
+ if (tmp_from < ud->rchan_cnt)
+ return -EINVAL;
+
+ if (tmp_from + cnt > ud->rflow_cnt)
+ return -EINVAL;
+
+ bitmap_or(tmp, ud->rflow_map, ud->rflow_map_reserved,
+ ud->rflow_cnt);
+
+ start = bitmap_find_next_zero_area(tmp,
+ ud->rflow_cnt,
+ tmp_from, cnt, 0);
+ if (start >= ud->rflow_cnt)
+ return -ENOMEM;
+
+ if (from >= 0 && start != from)
+ return -EEXIST;
+
+ bitmap_set(ud->rflow_map_reserved, start, cnt);
+ return start;
+}
+
+static int __udma_free_rflow_range(struct udma_dev *ud, int from, int cnt)
+{
+ if (from < ud->rchan_cnt)
+ return -EINVAL;
+ if (from + cnt > ud->rflow_cnt)
+ return -EINVAL;
+
+ bitmap_clear(ud->rflow_map_reserved, from, cnt);
+ return 0;
+}
+
+static struct udma_rflow *__udma_reserve_rflow(struct udma_dev *ud,
+ enum udma_tp_level tpl, int id)
+{
+ DECLARE_BITMAP(tmp, K3_UDMA_MAX_RFLOWS);
+
+ if (id >= 0) {
+ if (test_bit(id, ud->rflow_map)) {
+ dev_err(ud->dev, "rflow%d is in use\n", id);
+ return ERR_PTR(-ENOENT);
+ }
+ } else {
+ bitmap_or(tmp, ud->rflow_map, ud->rflow_map_reserved,
+ ud->rflow_cnt);
+
+ id = find_next_zero_bit(tmp, ud->rflow_cnt, ud->rchan_cnt);
+ if (id >= ud->rflow_cnt)
+ return ERR_PTR(-ENOENT);
+ }
+
+ set_bit(id, ud->rflow_map);
+ return &ud->rflows[id];
+}
+
+#define UDMA_RESERVE_RESOURCE(res) \
+static struct udma_##res *__udma_reserve_##res(struct udma_dev *ud, \
+ enum udma_tp_level tpl, \
+ int id) \
+{ \
+ if (id >= 0) { \
+ if (test_bit(id, ud->res##_map)) { \
+ dev_err(ud->dev, "res##%d is in use\n", id); \
+ return ERR_PTR(-ENOENT); \
+ } \
+ } else { \
+ int start; \
+ \
+ if (tpl >= ud->match_data->tpl_levels) \
+ tpl = ud->match_data->tpl_levels - 1; \
+ \
+ start = ud->match_data->level_start_idx[tpl]; \
+ \
+ id = find_next_zero_bit(ud->res##_map, ud->res##_cnt, \
+ start); \
+ if (id == ud->res##_cnt) { \
+ return ERR_PTR(-ENOENT); \
+ } \
+ } \
+ \
+ set_bit(id, ud->res##_map); \
+ return &ud->res##s[id]; \
+}
+
+UDMA_RESERVE_RESOURCE(tchan);
+UDMA_RESERVE_RESOURCE(rchan);
+
+static int udma_get_tchan(struct udma_chan *uc)
+{
+ struct udma_dev *ud = uc->ud;
+
+ if (uc->tchan) {
+ dev_dbg(ud->dev, "chan%d: already have tchan%d allocated\n",
+ uc->id, uc->tchan->id);
+ return 0;
+ }
+
+ uc->tchan = __udma_reserve_tchan(ud, uc->channel_tpl, -1);
+ if (IS_ERR(uc->tchan))
+ return PTR_ERR(uc->tchan);
+
+ if (udma_is_chan_running(uc)) {
+ dev_warn(ud->dev, "chan%d: tchan%d is running!\n", uc->id,
+ uc->tchan->id);
+ udma_stop(uc);
+ if (udma_is_chan_running(uc))
+ dev_err(ud->dev, "chan%d: won't stop!\n", uc->id);
+ }
+
+ return 0;
+}
+
+static int udma_get_rchan(struct udma_chan *uc)
+{
+ struct udma_dev *ud = uc->ud;
+
+ if (uc->rchan) {
+ dev_dbg(ud->dev, "chan%d: already have rchan%d allocated\n",
+ uc->id, uc->rchan->id);
+ return 0;
+ }
+
+ uc->rchan = __udma_reserve_rchan(ud, uc->channel_tpl, -1);
+ if (IS_ERR(uc->rchan))
+ return PTR_ERR(uc->rchan);
+
+ if (udma_is_chan_running(uc)) {
+ dev_warn(ud->dev, "chan%d: rchan%d is running!\n", uc->id,
+ uc->rchan->id);
+ udma_stop(uc);
+ if (udma_is_chan_running(uc))
+ dev_err(ud->dev, "chan%d: won't stop!\n", uc->id);
+ }
+
+ return 0;
+}
+
+static int udma_get_chan_pair(struct udma_chan *uc)
+{
+ struct udma_dev *ud = uc->ud;
+ int chan_id, end;
+
+ if ((uc->tchan && uc->rchan) && uc->tchan->id == uc->rchan->id) {
+ dev_info(ud->dev, "chan%d: already have %d pair allocated\n",
+ uc->id, uc->tchan->id);
+ return 0;
+ }
+
+ if (uc->tchan) {
+ dev_err(ud->dev, "chan%d: already have tchan%d allocated\n",
+ uc->id, uc->tchan->id);
+ return -EBUSY;
+ } else if (uc->rchan) {
+ dev_err(ud->dev, "chan%d: already have rchan%d allocated\n",
+ uc->id, uc->rchan->id);
+ return -EBUSY;
+ }
+
+ /* Can be optimized, but let's have it like this for now */
+ end = min(ud->tchan_cnt, ud->rchan_cnt);
+ for (chan_id = ud->match_data->level_start_idx[UDMA_TP_NORMAL];
+ chan_id < end; chan_id++) {
+ if (!test_bit(chan_id, ud->tchan_map) &&
+ !test_bit(chan_id, ud->rchan_map))
+ break;
+ }
+
+ if (chan_id == end)
+ return -ENOENT;
+
+ set_bit(chan_id, ud->tchan_map);
+ set_bit(chan_id, ud->rchan_map);
+ uc->tchan = &ud->tchans[chan_id];
+ uc->rchan = &ud->rchans[chan_id];
+
+ if (udma_is_chan_running(uc)) {
+ dev_warn(ud->dev, "chan%d: t/rchan%d pair is running!\n",
+ uc->id, chan_id);
+ udma_stop(uc);
+ if (udma_is_chan_running(uc))
+ dev_err(ud->dev, "chan%d: won't stop!\n", uc->id);
+ }
+
+ return 0;
+}
+
+static int udma_get_rflow(struct udma_chan *uc, int flow_id)
+{
+ struct udma_dev *ud = uc->ud;
+
+ if (uc->rflow) {
+ dev_dbg(ud->dev, "chan%d: already have rflow%d allocated\n",
+ uc->id, uc->rflow->id);
+ return 0;
+ }
+
+ if (!uc->rchan)
+ dev_warn(ud->dev, "chan%d: does not have rchan??\n", uc->id);
+
+ uc->rflow = __udma_reserve_rflow(ud, uc->channel_tpl, flow_id);
+ if (IS_ERR(uc->rflow))
+ return PTR_ERR(uc->rflow);
+
+ return 0;
+}
+
+static void udma_put_rchan(struct udma_chan *uc)
+{
+ struct udma_dev *ud = uc->ud;
+
+ if (uc->rchan) {
+ dev_dbg(ud->dev, "chan%d: put rchan%d\n", uc->id,
+ uc->rchan->id);
+ clear_bit(uc->rchan->id, ud->rchan_map);
+ uc->rchan = NULL;
+ }
+}
+
+static void udma_put_tchan(struct udma_chan *uc)
+{
+ struct udma_dev *ud = uc->ud;
+
+ if (uc->tchan) {
+ dev_dbg(ud->dev, "chan%d: put tchan%d\n", uc->id,
+ uc->tchan->id);
+ clear_bit(uc->tchan->id, ud->tchan_map);
+ uc->tchan = NULL;
+ }
+}
+
+static void udma_put_rflow(struct udma_chan *uc)
+{
+ struct udma_dev *ud = uc->ud;
+
+ if (uc->rflow) {
+ dev_dbg(ud->dev, "chan%d: put rflow%d\n", uc->id,
+ uc->rflow->id);
+ clear_bit(uc->rflow->id, ud->rflow_map);
+ uc->rflow = NULL;
+ }
+}
+
+static void udma_free_tx_resources(struct udma_chan *uc)
+{
+ if (!uc->tchan)
+ return;
+
+ k3_nav_ringacc_ring_free(uc->tchan->t_ring);
+ k3_nav_ringacc_ring_free(uc->tchan->tc_ring);
+ uc->tchan->t_ring = NULL;
+ uc->tchan->tc_ring = NULL;
+
+ udma_put_tchan(uc);
+}
+
+static int udma_alloc_tx_resources(struct udma_chan *uc)
+{
+ struct k3_nav_ring_cfg ring_cfg;
+ struct udma_dev *ud = uc->ud;
+ int ret;
+
+ ret = udma_get_tchan(uc);
+ if (ret)
+ return ret;
+
+ uc->tchan->t_ring = k3_nav_ringacc_request_ring(ud->ringacc,
+ uc->tchan->id, 0);
+ if (!uc->tchan->t_ring) {
+ ret = -EBUSY;
+ goto err_tx_ring;
+ }
+
+ uc->tchan->tc_ring = k3_nav_ringacc_request_ring(ud->ringacc, -1, 0);
+ if (!uc->tchan->tc_ring) {
+ ret = -EBUSY;
+ goto err_txc_ring;
+ }
+
+ memset(&ring_cfg, 0, sizeof(ring_cfg));
+ ring_cfg.size = 16;
+ ring_cfg.elm_size = K3_NAV_RINGACC_RING_ELSIZE_8;
+ ring_cfg.mode = K3_NAV_RINGACC_RING_MODE_MESSAGE;
+
+ ret = k3_nav_ringacc_ring_cfg(uc->tchan->t_ring, &ring_cfg);
+ ret |= k3_nav_ringacc_ring_cfg(uc->tchan->tc_ring, &ring_cfg);
+
+ if (ret)
+ goto err_ringcfg;
+
+ return 0;
+
+err_ringcfg:
+ k3_nav_ringacc_ring_free(uc->tchan->tc_ring);
+ uc->tchan->tc_ring = NULL;
+err_txc_ring:
+ k3_nav_ringacc_ring_free(uc->tchan->t_ring);
+ uc->tchan->t_ring = NULL;
+err_tx_ring:
+ udma_put_tchan(uc);
+
+ return ret;
+}
+
+static void udma_free_rx_resources(struct udma_chan *uc)
+{
+ if (!uc->rchan)
+ return;
+
+ if (uc->dir != DMA_MEM_TO_MEM) {
+ k3_nav_ringacc_ring_free(uc->rchan->fd_ring);
+ k3_nav_ringacc_ring_free(uc->rchan->r_ring);
+ uc->rchan->fd_ring = NULL;
+ uc->rchan->r_ring = NULL;
+
+ udma_put_rflow(uc);
+ }
+
+ udma_put_rchan(uc);
+}
+
+static int udma_alloc_rx_resources(struct udma_chan *uc)
+{
+ struct k3_nav_ring_cfg ring_cfg;
+ struct udma_dev *ud = uc->ud;
+ int fd_ring_id;
+ int ret;
+
+ ret = udma_get_rchan(uc);
+ if (ret)
+ return ret;
+
+ /* For MEM_TO_MEM we don't need rflow or rings */
+ if (uc->dir == DMA_MEM_TO_MEM)
+ return 0;
+
+ ret = udma_get_rflow(uc, uc->rchan->id);
+ if (ret) {
+ ret = -EBUSY;
+ goto err_rflow;
+ }
+
+ fd_ring_id = ud->tchan_cnt + ud->echan_cnt + uc->rchan->id;
+ uc->rchan->fd_ring = k3_nav_ringacc_request_ring(ud->ringacc,
+ fd_ring_id, 0);
+ if (!uc->rchan->fd_ring) {
+ ret = -EBUSY;
+ goto err_rx_ring;
+ }
+
+ uc->rchan->r_ring = k3_nav_ringacc_request_ring(ud->ringacc, -1, 0);
+ if (!uc->rchan->r_ring) {
+ ret = -EBUSY;
+ goto err_rxc_ring;
+ }
+
+ memset(&ring_cfg, 0, sizeof(ring_cfg));
+ ring_cfg.size = 16;
+ ring_cfg.elm_size = K3_NAV_RINGACC_RING_ELSIZE_8;
+ ring_cfg.mode = K3_NAV_RINGACC_RING_MODE_MESSAGE;
+
+ ret = k3_nav_ringacc_ring_cfg(uc->rchan->fd_ring, &ring_cfg);
+ ret |= k3_nav_ringacc_ring_cfg(uc->rchan->r_ring, &ring_cfg);
+
+ if (ret)
+ goto err_ringcfg;
+
+ return 0;
+
+err_ringcfg:
+ k3_nav_ringacc_ring_free(uc->rchan->r_ring);
+ uc->rchan->r_ring = NULL;
+err_rxc_ring:
+ k3_nav_ringacc_ring_free(uc->rchan->fd_ring);
+ uc->rchan->fd_ring = NULL;
+err_rx_ring:
+ udma_put_rflow(uc);
+err_rflow:
+ udma_put_rchan(uc);
+
+ return ret;
+}
+
+static int udma_alloc_chan_resources(struct dma_chan *chan)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ struct udma_dev *ud = to_udma_dev(chan->device);
+ struct udma_tisci_rm *tisci_rm = &ud->tisci_rm;
+ const struct ti_sci_rm_udmap_ops *tisci_ops = tisci_rm->tisci_udmap_ops;
+ struct udma_tchan *tchan;
+ struct udma_rchan *rchan;
+ int ret;
+
+ if (uc->pkt_mode || uc->dir == DMA_MEM_TO_MEM) {
+ uc->use_dma_pool = true;
+ /* in case of MEM_TO_MEM we have maximum of two TRs */
+ if (uc->dir == DMA_MEM_TO_MEM)
+ uc->hdesc_size = udma_calc_trdesc_size(
+ sizeof(struct cppi50_tr_req_type15), 2);
+ }
+
+ if (uc->use_dma_pool) {
+ uc->hdesc_pool = dma_pool_create(uc->name, ud->ddev.dev,
+ uc->hdesc_size, uc->desc_align,
+ 0);
+ if (!uc->hdesc_pool) {
+ dev_err(ud->ddev.dev,
+ "Descriptor pool allocation failed\n");
+ uc->use_dma_pool = false;
+ return -ENOMEM;
+ }
+ }
+
+ pm_runtime_get_sync(ud->ddev.dev);
+
+ /*
+ * Make sure that the completion is in a known state:
+ * No teardown, the channel is idle
+ */
+ reinit_completion(&uc->teardown_completed);
+ complete_all(&uc->teardown_completed);
+ uc->state = UDMA_CHAN_IS_IDLE;
+
+ switch (uc->dir) {
+ case DMA_MEM_TO_MEM:
+ /* Non synchronized - mem to mem type of transfer */
+ dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-MEM\n", __func__,
+ uc->id);
+
+ ret = udma_get_chan_pair(uc);
+ if (ret)
+ return ret;
+
+ ret = udma_alloc_tx_resources(uc);
+ if (ret)
+ return ret;
+
+ ret = udma_alloc_rx_resources(uc);
+ if (ret) {
+ udma_free_tx_resources(uc);
+ return ret;
+ }
+
+ uc->src_thread = ud->psil_base + uc->tchan->id;
+ uc->dst_thread = (ud->psil_base + uc->rchan->id) | 0x8000;
+
+ break;
+ case DMA_MEM_TO_DEV:
+ /* Slave transfer synchronized - mem to dev (TX) trasnfer */
+ dev_dbg(uc->ud->dev, "%s: chan%d as MEM-to-DEV\n", __func__,
+ uc->id);
+
+ ret = udma_alloc_tx_resources(uc);
+ if (ret) {
+ uc->slave_thread_id = -1;
+ return ret;
+ }
+
+ uc->src_thread = ud->psil_base + uc->tchan->id;
+ uc->dst_thread = uc->slave_thread_id;
+ if (!(uc->dst_thread & 0x8000))
+ uc->dst_thread |= 0x8000;
+
+ break;
+ case DMA_DEV_TO_MEM:
+ /* Slave transfer synchronized - dev to mem (RX) trasnfer */
+ dev_dbg(uc->ud->dev, "%s: chan%d as DEV-to-MEM\n", __func__,
+ uc->id);
+
+ ret = udma_alloc_rx_resources(uc);
+ if (ret) {
+ uc->slave_thread_id = -1;
+ return ret;
+ }
+
+ uc->src_thread = uc->slave_thread_id;
+ uc->dst_thread = (ud->psil_base + uc->rchan->id) | 0x8000;
+
+ break;
+ default:
+ /* Can not happen */
+ dev_err(uc->ud->dev, "%s: chan%d invalid direction (%u)\n",
+ __func__, uc->id, uc->dir);
+ return -EINVAL;
+ }
+
+ tchan = uc->tchan;
+ rchan = uc->rchan;
+
+ /*
+ * Configure Tx and Rx channel type to:
+ * Third Party DMA control transfers using pass by reference rings
+ */
+ if (uc->dir == DMA_MEM_TO_MEM) {
+ /* Non synchronized - mem to mem type of transfer */
+ int tc_ring = k3_nav_ringacc_get_ring_id(tchan->tc_ring);
+ struct ti_sci_msg_rm_udmap_tx_ch_cfg req_tx = { 0 };
+ struct ti_sci_msg_rm_udmap_rx_ch_cfg req_rx = { 0 };
+
+ req_tx.valid_params =
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_EINFO_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_PSWORDS_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_SUPR_TDPKT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID;
+
+ req_tx.nav_id = tisci_rm->tisci_dev_id;
+ req_tx.index = tchan->id;
+ req_tx.tx_pause_on_err = 0;
+ req_tx.tx_filt_einfo = 0;
+ req_tx.tx_filt_pswords = 0;
+ req_tx.tx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_3RDP_BCOPY_PBRR;
+ req_tx.tx_supr_tdpkt = 0;
+ req_tx.tx_fetch_size = sizeof(struct cppi50_tr_req_desc) >> 2;
+ req_tx.txcq_qnum = tc_ring;
+
+ ret = tisci_ops->tx_ch_cfg(tisci_rm->tisci, &req_tx);
+ if (ret) {
+ dev_err(ud->dev, "tchan%d cfg failed %d\n",
+ tchan->id, ret);
+ goto err_res_free;
+ }
+
+ req_rx.valid_params =
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_SHORT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_LONG_VALID;
+
+ req_rx.nav_id = tisci_rm->tisci_dev_id;
+ req_rx.index = rchan->id;
+ req_rx.rx_fetch_size = sizeof(struct cppi50_tr_req_desc) >> 2;
+ req_rx.rxcq_qnum = tc_ring;
+ req_rx.rx_pause_on_err = 0;
+ req_rx.rx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_3RDP_BCOPY_PBRR;
+ req_rx.rx_ignore_short = 0;
+ req_rx.rx_ignore_long = 0;
+
+ ret = tisci_ops->rx_ch_cfg(tisci_rm->tisci, &req_rx);
+ if (ret) {
+ dev_err(ud->dev, "rchan%d alloc failed %d\n",
+ rchan->id, ret);
+ goto err_res_free;
+ }
+
+ uc->irq_ra_tisci = k3_nav_ringacc_get_tisci_dev_id(
+ tchan->tc_ring);
+ uc->irq_ra_idx = tc_ring;
+ uc->irq_udma_idx = tchan->id;
+ } else {
+ /* Slave transfer */
+ u32 mode, fetch_size;
+
+ if (uc->pkt_mode) {
+ mode = TI_SCI_RM_UDMAP_CHAN_TYPE_PKT_PBRR;
+ fetch_size = knav_udmap_hdesc_calc_size(uc->needs_epib,
+ uc->psd_size,
+ 0);
+ } else {
+ mode = TI_SCI_RM_UDMAP_CHAN_TYPE_3RDP_PBRR;
+ fetch_size = sizeof(struct cppi50_tr_req_desc);
+ }
+
+ if (uc->dir == DMA_MEM_TO_DEV) {
+ /* TX */
+ int tc_ring = k3_nav_ringacc_get_ring_id(
+ tchan->tc_ring);
+ struct ti_sci_msg_rm_udmap_tx_ch_cfg req_tx = { 0 };
+
+ req_tx.valid_params =
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_EINFO_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_PSWORDS_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_SUPR_TDPKT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID;
+
+ req_tx.nav_id = tisci_rm->tisci_dev_id;
+ req_tx.index = tchan->id;
+ req_tx.tx_pause_on_err = 0;
+ req_tx.tx_filt_einfo = 0;
+ req_tx.tx_filt_pswords = 0;
+ req_tx.tx_chan_type = mode;
+ req_tx.tx_supr_tdpkt = 0;
+ req_tx.tx_fetch_size = fetch_size >> 2;
+ req_tx.txcq_qnum = tc_ring;
+
+ ret = tisci_ops->tx_ch_cfg(tisci_rm->tisci, &req_tx);
+ if (ret) {
+ dev_err(ud->dev, "tchan%d cfg failed %d\n",
+ tchan->id, ret);
+ goto err_res_free;
+ }
+
+ uc->irq_ra_tisci = k3_nav_ringacc_get_tisci_dev_id(
+ tchan->tc_ring);
+ uc->irq_ra_idx = tc_ring;
+ uc->irq_udma_idx = tchan->id;
+ } else {
+ /* RX */
+ int fd_ring = k3_nav_ringacc_get_ring_id(
+ rchan->fd_ring);
+ int rx_ring = k3_nav_ringacc_get_ring_id(rchan->r_ring);
+ struct ti_sci_msg_rm_udmap_rx_ch_cfg req_rx = { 0 };
+ struct ti_sci_msg_rm_udmap_flow_cfg flow_req = { 0 };
+
+ req_rx.valid_params =
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_SHORT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_IGNORE_LONG_VALID;
+
+ req_rx.nav_id = tisci_rm->tisci_dev_id;
+ req_rx.index = rchan->id;
+ req_rx.rx_fetch_size = fetch_size >> 2;
+ req_rx.rxcq_qnum = rx_ring;
+ req_rx.rx_pause_on_err = 0;
+ req_rx.rx_chan_type = mode;
+ req_rx.rx_ignore_short = 0;
+ req_rx.rx_ignore_long = 0;
+
+ ret = tisci_ops->rx_ch_cfg(tisci_rm->tisci, &req_rx);
+ if (ret) {
+ dev_err(ud->dev, "rchan%d cfg failed %d\n",
+ rchan->id, ret);
+ goto err_res_free;
+ }
+
+ flow_req.valid_params =
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_EINFO_PRESENT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_PSINFO_PRESENT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_ERROR_HANDLING_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DESC_TYPE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_HI_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_LO_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_HI_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_LO_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ0_SZ0_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ1_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ2_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ3_QNUM_VALID;
+
+ flow_req.nav_id = tisci_rm->tisci_dev_id;
+ flow_req.flow_index = rchan->id;
+
+ if (uc->needs_epib)
+ flow_req.rx_einfo_present = 1;
+ else
+ flow_req.rx_einfo_present = 0;
+ if (uc->psd_size)
+ flow_req.rx_psinfo_present = 1;
+ else
+ flow_req.rx_psinfo_present = 0;
+ flow_req.rx_error_handling = 1;
+ flow_req.rx_desc_type = 0;
+ flow_req.rx_dest_qnum = rx_ring;
+ flow_req.rx_src_tag_hi_sel = 2;
+ flow_req.rx_src_tag_lo_sel = 4;
+ flow_req.rx_dest_tag_hi_sel = 5;
+ flow_req.rx_dest_tag_lo_sel = 4;
+ flow_req.rx_fdq0_sz0_qnum = fd_ring;
+ flow_req.rx_fdq1_qnum = fd_ring;
+ flow_req.rx_fdq2_qnum = fd_ring;
+ flow_req.rx_fdq3_qnum = fd_ring;
+
+ ret = tisci_ops->rx_flow_cfg(tisci_rm->tisci,
+ &flow_req);
+
+ if (ret) {
+ dev_err(ud->dev, "flow%d config failed: %d\n",
+ rchan->id, ret);
+ goto err_chan_free;
+ }
+
+ uc->irq_ra_tisci = k3_nav_ringacc_get_tisci_dev_id(
+ rchan->r_ring);
+ uc->irq_ra_idx = rx_ring;
+ uc->irq_udma_idx = 0x2000 + rchan->id;
+ }
+ }
+
+ /* PSI-L pairing */
+ ret = navss_psil_pair(ud, uc->src_thread, uc->dst_thread);
+ if (ret)
+ goto err_chan_free;
+
+ uc->psil_paired = true;
+
+ /* Get the interrupts... */
+ uc->irq_num_ring = ti_sci_inta_register_event(ud->dev, uc->irq_ra_tisci,
+ uc->irq_ra_idx, 0,
+ IRQF_TRIGGER_HIGH);
+ if (uc->irq_num_ring <= 0) {
+ dev_err(ud->dev, "Failed to get ring irq (index: %u) %d\n",
+ uc->irq_ra_idx, uc->irq_num_ring);
+ ret = -EINVAL;
+ goto err_psi_free;
+ }
+
+ uc->irq_num_udma = ti_sci_inta_register_event(ud->dev,
+ tisci_rm->tisci_dev_id,
+ uc->irq_udma_idx, 0,
+ IRQF_TRIGGER_HIGH);
+ if (uc->irq_num_udma <= 0) {
+ dev_err(ud->dev, "Failed to get udma irq (index: %u) %d\n",
+ uc->irq_udma_idx, uc->irq_num_udma);
+
+ ti_sci_inta_unregister_event(ud->dev, uc->irq_ra_tisci,
+ uc->irq_ra_idx, uc->irq_num_ring);
+
+ ret = -EINVAL;
+ goto err_psi_free;
+ }
+
+ ret = request_irq(uc->irq_num_ring, udma_ring_irq_handler, 0, uc->name,
+ uc);
+ if (ret) {
+ dev_err(ud->dev, "%s: chan%d: Failed to request ring irq\n",
+ __func__, uc->id);
+ goto err_irq_free;
+ }
+
+ ret = request_irq(uc->irq_num_udma, udma_udma_irq_handler, 0, uc->name,
+ uc);
+ if (ret) {
+ dev_err(ud->dev, "%s: chan%d: Failed to request UDMA irq\n",
+ __func__, uc->id);
+ free_irq(uc->irq_num_ring, uc);
+ goto err_irq_free;
+ }
+
+ udma_reset_rings(uc);
+
+ return 0;
+
+err_irq_free:
+ ti_sci_inta_unregister_event(ud->dev, uc->irq_ra_tisci, uc->irq_ra_idx,
+ uc->irq_num_ring);
+ uc->irq_num_ring = 0;
+
+ ti_sci_inta_unregister_event(ud->dev, tisci_rm->tisci_dev_id,
+ uc->irq_udma_idx, uc->irq_num_udma);
+ uc->irq_num_udma = 0;
+err_psi_free:
+ navss_psil_unpair(ud, uc->src_thread, uc->dst_thread);
+ uc->psil_paired = false;
+err_chan_free:
+err_res_free:
+ udma_free_tx_resources(uc);
+ udma_free_rx_resources(uc);
+ uc->slave_thread_id = -1;
+
+ if (uc->use_dma_pool) {
+ dma_pool_destroy(uc->hdesc_pool);
+ uc->use_dma_pool = false;
+ }
+
+ return ret;
+}
+
+static int udma_slave_config(struct dma_chan *chan,
+ struct dma_slave_config *cfg)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+
+ memcpy(&uc->cfg, cfg, sizeof(uc->cfg));
+
+ return 0;
+}
+
+static struct udma_desc *udma_alloc_tr_desc(struct udma_chan *uc,
+ size_t tr_size, int tr_count,
+ enum dma_transfer_direction dir)
+{
+ struct udma_hwdesc *hwdesc;
+ struct cppi50_tr_req_desc *tr_desc;
+ struct udma_desc *d;
+ u32 tr_nominal_size;
+
+ switch (tr_size) {
+ case 16:
+ tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_16;
+ break;
+ case 32:
+ tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_32;
+ break;
+ case 64:
+ tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_64;
+ break;
+ case 128:
+ tr_nominal_size = CPPI50_TRDESC_W1_TR_SIZE_128;
+ break;
+ default:
+ dev_err(uc->ud->dev, "Unsupported TR size of %zu\n", tr_size);
+ return NULL;
+ }
+
+ /* We have only one descriptor containing multiple TRs */
+ d = kzalloc(sizeof(*d) + sizeof(d->hwdesc[0]), GFP_ATOMIC);
+ if (!d)
+ return NULL;
+
+ d->sglen = tr_count;
+
+ d->hwdesc_count = 1;
+ hwdesc = &d->hwdesc[0];
+
+ /* Allocate memory for DMA ring descriptor */
+ if (uc->use_dma_pool) {
+ hwdesc->cppi5_desc_size = uc->hdesc_size;
+ hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool,
+ GFP_ATOMIC,
+ &hwdesc->cppi5_desc_paddr);
+ } else {
+ hwdesc->cppi5_desc_size = udma_calc_trdesc_size(tr_size,
+ tr_count);
+ hwdesc->cppi5_desc_vaddr = dma_zalloc_coherent(uc->ud->dev,
+ hwdesc->cppi5_desc_size,
+ &hwdesc->cppi5_desc_paddr,
+ GFP_ATOMIC);
+ }
+
+ if (!hwdesc->cppi5_desc_vaddr) {
+ kfree(d);
+ return NULL;
+ }
+
+ /* Start of the TR req records */
+ hwdesc->tr_req_base = hwdesc->cppi5_desc_vaddr + tr_size;
+ /* Start address of the TR response array */
+ hwdesc->tr_resp_base = hwdesc->tr_req_base + tr_size * tr_count;
+
+ tr_desc = hwdesc->cppi5_desc_vaddr;
+ tr_desc->packet_info[0] = CPPI50_TRDESC_W0_LAST_ENTRY(tr_count - 1) |
+ CPPI50_TRDESC_W0_TYPE;
+ if (uc->cyclic)
+ tr_desc->packet_info[0] |= CPPI50_TRDESC_W0_RELOAD_CNT(0x1ff);
+
+ /* Flow and Packed ID */
+ tr_desc->packet_info[1] = tr_nominal_size |
+ CPPI50_TRDESC_W1_PACKETID(uc->id) |
+ CPPI50_TRDESC_W1_FLOWID(0x3fff);
+
+ if (dir == DMA_DEV_TO_MEM)
+ tr_desc->packet_info[2] = k3_nav_ringacc_get_ring_id(
+ uc->rchan->r_ring);
+ else
+ tr_desc->packet_info[2] = k3_nav_ringacc_get_ring_id(
+ uc->tchan->tc_ring);
+
+ return d;
+}
+
+static struct udma_desc *udma_prep_slave_sg_tr(
+ struct udma_chan *uc, struct scatterlist *sgl, unsigned int sglen,
+ enum dma_transfer_direction dir, unsigned long tx_flags, void *context)
+{
+ enum dma_slave_buswidth dev_width;
+ struct scatterlist *sgent;
+ struct udma_desc *d;
+ size_t tr_size;
+ struct cppi50_tr_req_type1 *tr_req = NULL;
+ unsigned int i;
+ u8 elsize;
+ u32 burst;
+
+ if (dir == DMA_DEV_TO_MEM) {
+ dev_width = uc->cfg.src_addr_width;
+ burst = uc->cfg.src_maxburst;
+ } else if (dir == DMA_MEM_TO_DEV) {
+ dev_width = uc->cfg.dst_addr_width;
+ burst = uc->cfg.dst_maxburst;
+ } else {
+ dev_err(uc->ud->dev, "%s: bad direction?\n", __func__);
+ return NULL;
+ }
+
+ /* Bus width translates to the element size (ES) */
+ switch (dev_width) {
+ case DMA_SLAVE_BUSWIDTH_1_BYTE:
+ elsize = UDMA_ELSIZE_8;
+ break;
+ case DMA_SLAVE_BUSWIDTH_2_BYTES:
+ elsize = UDMA_ELSIZE_16;
+ break;
+ case DMA_SLAVE_BUSWIDTH_3_BYTES:
+ elsize = UDMA_ELSIZE_24;
+ break;
+ case DMA_SLAVE_BUSWIDTH_4_BYTES:
+ elsize = UDMA_ELSIZE_32;
+ break;
+ case DMA_SLAVE_BUSWIDTH_8_BYTES:
+ elsize = UDMA_ELSIZE_64;
+ break;
+ default: /* not reached */
+ return NULL;
+ }
+
+ if (!burst)
+ burst = 1;
+
+ /* Now allocate and setup the descriptor. */
+ tr_size = sizeof(struct cppi50_tr_req_type1);
+ d = udma_alloc_tr_desc(uc, tr_size, sglen, dir);
+ if (!d)
+ return NULL;
+
+ d->sglen = sglen;
+
+ tr_req = (struct cppi50_tr_req_type1 *)d->hwdesc[0].tr_req_base;
+ for_each_sg(sgl, sgent, sglen, i) {
+ d->residue += sg_dma_len(sgent);
+ tr_req[i].flags = CPPI50_TR_FLAGS_TYPE(1);
+ tr_req[i].addr = sg_dma_address(sgent);
+ tr_req[i].icnt0 = burst * elsize_bytes[elsize];
+ tr_req[i].dim1 = burst * elsize_bytes[elsize];
+ tr_req[i].icnt1 = sg_dma_len(sgent) / tr_req[i].icnt0;
+
+ tr_req[i].flags |= CPPI50_TR_FLAGS_SUPR_EVT;
+ }
+
+ tr_req[i - 1].flags |= CPPI50_TR_FLAGS_EOP;
+
+ return d;
+}
+
+static inline int udma_configure_statictr(struct udma_chan *uc,
+ struct udma_desc *d, u8 elsize,
+ u16 elcnt)
+{
+ if (!uc->static_tr_type)
+ return 0;
+
+ d->static_tr.elsize = elsize;
+ d->static_tr.elcnt = elcnt;
+ if (uc->pkt_mode) {
+ unsigned int div = elsize_bytes[elsize] * elcnt;
+
+ if (uc->cyclic)
+ d->static_tr.bstcnt = d->residue / d->sglen / div;
+ else
+ d->static_tr.bstcnt = d->residue / div;
+
+ if (uc->dir == DMA_DEV_TO_MEM && d->static_tr.bstcnt > 0xfff)
+ return -EINVAL;
+ } else {
+ d->static_tr.bstcnt = 0;
+ }
+
+ return 0;
+}
+
+static struct udma_desc *udma_prep_slave_sg_pkt(
+ struct udma_chan *uc, struct scatterlist *sgl, unsigned int sglen,
+ enum dma_transfer_direction dir, unsigned long tx_flags, void *context)
+{
+ struct scatterlist *sgent;
+ struct knav_udmap_host_desc_t *h_desc = NULL;
+ struct udma_desc *d;
+ u32 ring_id;
+ unsigned int i;
+
+ d = kzalloc(sizeof(*d) + sglen * sizeof(d->hwdesc[0]), GFP_ATOMIC);
+ if (!d)
+ return NULL;
+
+ if (dir == DMA_DEV_TO_MEM && sglen > 1) {
+ void *buffer;
+ int ret;
+ size_t total_len = 0;
+
+ /* Count the total length of the receive SG buffer */
+ for_each_sg(sgl, sgent, sglen, i)
+ total_len += sg_dma_len(sgent);
+
+ buffer = kzalloc(total_len, GFP_ATOMIC);
+ if (!buffer) {
+ kfree(d);
+ return NULL;
+ }
+
+ sg_init_table(&d->rx_sg_wa.single_sg, 1);
+ sg_set_buf(&d->rx_sg_wa.single_sg, buffer, total_len);
+ ret = dma_map_sg(uc->ud->dev, &d->rx_sg_wa.single_sg, 1,
+ DMA_FROM_DEVICE);
+ if (ret != 1) {
+ dev_err(uc->ud->dev,
+ "mapping of temp buffer error (%d)\n", ret);
+ kfree(buffer);
+ kfree(d);
+ return NULL;
+ }
+
+ d->rx_sg_wa.in_use = true;
+
+ d->rx_sg_wa.sgl = sgl;
+ d->rx_sg_wa.sglen = sglen;
+ d->rx_sg_wa.total_len = total_len;
+
+ sgl = &d->rx_sg_wa.single_sg;
+ sglen = 1;
+ }
+
+ d->sglen = sglen;
+ d->hwdesc_count = sglen;
+
+ if (dir == DMA_DEV_TO_MEM)
+ ring_id = k3_nav_ringacc_get_ring_id(uc->rchan->r_ring);
+ else
+ ring_id = k3_nav_ringacc_get_ring_id(uc->tchan->tc_ring);
+
+ for_each_sg(sgl, sgent, sglen, i) {
+ struct udma_hwdesc *hwdesc = &d->hwdesc[i];
+ dma_addr_t sg_addr = sg_dma_address(sgent);
+ struct knav_udmap_host_desc_t *desc;
+ size_t sg_len = sg_dma_len(sgent);
+
+ hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool,
+ GFP_ATOMIC,
+ &hwdesc->cppi5_desc_paddr);
+ if (!hwdesc->cppi5_desc_vaddr) {
+ dev_err(uc->ud->dev,
+ "descriptor%d allocation failed\n", i);
+
+ udma_free_hwdesc(&d->vd);
+ kfree(d);
+ return NULL;
+
+ }
+
+ d->residue += sg_len;
+ hwdesc->cppi5_desc_size = uc->hdesc_size;
+ desc = hwdesc->cppi5_desc_vaddr;
+
+ if (i == 0) {
+ knav_udmap_hdesc_init(desc, 0, 0);
+ /* Flow and Packed ID ??? */
+ knav_udmap_hdesc_set_pktids(&desc->hdr, uc->id, 0x3fff);
+ knav_udmap_desc_set_retpolicy(&desc->hdr, 0, ring_id);
+ } else {
+ knav_udmap_hdesc_reset_hbdesc(desc);
+ knav_udmap_desc_set_retpolicy(&desc->hdr, 0, 0xffff);
+ }
+
+ /* attach the sg buffer to the descriptor */
+ knav_udmap_hdesc_attach_buf(desc,
+ sg_addr, sg_len,
+ sg_addr, sg_len);
+
+ /* Attach link as host buffer descriptor */
+ if (h_desc)
+ knav_udmap_hdesc_link_hbdesc(h_desc,
+ hwdesc->cppi5_desc_paddr);
+
+ h_desc = desc;
+ }
+
+ if (d->residue >= SZ_4M) {
+ dev_err(uc->ud->dev,
+ "%s: Transfer size %u is over the supported 4M range\n",
+ __func__, d->residue);
+ udma_free_hwdesc(&d->vd);
+ kfree(d);
+ return NULL;
+ }
+
+ h_desc = d->hwdesc[0].cppi5_desc_vaddr;
+ knav_udmap_hdesc_set_pktlen(h_desc, d->residue);
+
+ return d;
+}
+
+static int udma_attach_metadata(struct dma_async_tx_descriptor *desc,
+ void *data, size_t len)
+{
+ struct udma_desc *d = to_udma_desc(desc);
+ struct udma_chan *uc = to_udma_chan(desc->chan);
+ struct knav_udmap_host_desc_t *h_desc;
+ u32 psd_size = len;
+ u32 flags = 0;
+
+ if (!uc->pkt_mode || !uc->metadata_size)
+ return -ENOTSUPP;
+
+ if (!data || len > uc->metadata_size)
+ return -EINVAL;
+
+ if (uc->needs_epib && len < 16)
+ return -EINVAL;
+
+ h_desc = d->hwdesc[0].cppi5_desc_vaddr;
+ if (d->dir == DMA_MEM_TO_DEV)
+ memcpy(h_desc->epib, data, len);
+
+ if (uc->needs_epib)
+ psd_size -= 16;
+
+ d->metadata = data;
+ d->metadata_size = len;
+ if (uc->needs_epib)
+ flags |= KNAV_UDMAP_INFO0_HDESC_EPIB_PRESENT;
+
+ knav_udmap_hdesc_update_flags(h_desc, flags);
+ knav_udmap_hdesc_update_psdata_size(h_desc, psd_size);
+
+ return 0;
+}
+
+static void *udma_get_metadata_ptr(struct dma_async_tx_descriptor *desc,
+ size_t *payload_len, size_t *max_len)
+{
+ struct udma_desc *d = to_udma_desc(desc);
+ struct udma_chan *uc = to_udma_chan(desc->chan);
+ struct knav_udmap_host_desc_t *h_desc;
+
+ if (!uc->pkt_mode || !uc->metadata_size)
+ return ERR_PTR(-ENOTSUPP);
+
+ h_desc = d->hwdesc[0].cppi5_desc_vaddr;
+
+ *max_len = uc->metadata_size;
+
+ *payload_len = knav_udmap_desc_is_epib_present(&h_desc->hdr) ? 16 : 0;
+ *payload_len += knav_udmap_hdesc_get_psdata_size(h_desc);
+
+ return h_desc->epib;
+}
+
+static int udma_set_metadata_len(struct dma_async_tx_descriptor *desc,
+ size_t payload_len)
+{
+ struct udma_desc *d = to_udma_desc(desc);
+ struct udma_chan *uc = to_udma_chan(desc->chan);
+ struct knav_udmap_host_desc_t *h_desc;
+ u32 psd_size = payload_len;
+ u32 flags = 0;
+
+ if (!uc->pkt_mode || !uc->metadata_size)
+ return -ENOTSUPP;
+
+ if (payload_len > uc->metadata_size)
+ return -EINVAL;
+
+ if (uc->needs_epib && payload_len < 16)
+ return -EINVAL;
+
+ h_desc = d->hwdesc[0].cppi5_desc_vaddr;
+
+ if (uc->needs_epib) {
+ psd_size -= 16;
+ flags |= KNAV_UDMAP_INFO0_HDESC_EPIB_PRESENT;
+ }
+
+ knav_udmap_hdesc_update_flags(h_desc, flags);
+ knav_udmap_hdesc_update_psdata_size(h_desc, psd_size);
+
+ return 0;
+}
+
+static struct dma_descriptor_metadata_ops metadata_ops = {
+ .attach = udma_attach_metadata,
+ .get_ptr = udma_get_metadata_ptr,
+ .set_len = udma_set_metadata_len,
+};
+
+static struct dma_async_tx_descriptor *udma_prep_slave_sg(
+ struct dma_chan *chan, struct scatterlist *sgl, unsigned int sglen,
+ enum dma_transfer_direction dir, unsigned long tx_flags, void *context)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ enum dma_slave_buswidth dev_width;
+ struct udma_desc *d;
+ u8 elsize;
+ u32 burst;
+
+ if (dir != uc->dir) {
+ dev_err(chan->device->dev,
+ "%s: chan%d is for %s, not supporting %s\n",
+ __func__, uc->id, udma_get_dir_text(uc->dir),
+ udma_get_dir_text(dir));
+ return NULL;
+ }
+
+ if (dir == DMA_DEV_TO_MEM) {
+ dev_width = uc->cfg.src_addr_width;
+ burst = uc->cfg.src_maxburst;
+ } else if (dir == DMA_MEM_TO_DEV) {
+ dev_width = uc->cfg.dst_addr_width;
+ burst = uc->cfg.dst_maxburst;
+ } else {
+ dev_err(chan->device->dev, "%s: bad direction?\n", __func__);
+ return NULL;
+ }
+
+ /* Bus width translates to the element size (ES) */
+ switch (dev_width) {
+ case DMA_SLAVE_BUSWIDTH_1_BYTE:
+ elsize = UDMA_ELSIZE_8;
+ break;
+ case DMA_SLAVE_BUSWIDTH_2_BYTES:
+ elsize = UDMA_ELSIZE_16;
+ break;
+ case DMA_SLAVE_BUSWIDTH_3_BYTES:
+ elsize = UDMA_ELSIZE_24;
+ break;
+ case DMA_SLAVE_BUSWIDTH_4_BYTES:
+ elsize = UDMA_ELSIZE_32;
+ break;
+ case DMA_SLAVE_BUSWIDTH_8_BYTES:
+ elsize = UDMA_ELSIZE_64;
+ break;
+ default: /* not reached */
+ return NULL;
+ }
+
+ if (!burst)
+ burst = 1;
+
+ if (uc->pkt_mode)
+ d = udma_prep_slave_sg_pkt(uc, sgl, sglen, dir, tx_flags,
+ context);
+ else
+ d = udma_prep_slave_sg_tr(uc, sgl, sglen, dir, tx_flags,
+ context);
+
+ if (!d)
+ return NULL;
+
+ d->dir = dir;
+ d->desc_idx = 0;
+ d->tr_idx = 0;
+
+ /* static TR for remote PDMA */
+ if (udma_configure_statictr(uc, d, elsize, burst)) {
+ dev_err(uc->ud->dev,
+ "%s: StaticTR Z is limted to maximum 4095 (%u)\n",
+ __func__, d->static_tr.bstcnt);
+
+ udma_free_hwdesc(&d->vd);
+ kfree(d);
+ return NULL;
+ }
+
+ if (uc->metadata_size)
+ d->vd.tx.metadata_ops = &metadata_ops;
+
+ return vchan_tx_prep(&uc->vc, &d->vd, tx_flags);
+}
+
+static struct udma_desc *udma_prep_dma_cyclic_tr(
+ struct udma_chan *uc, dma_addr_t buf_addr, size_t buf_len,
+ size_t period_len, enum dma_transfer_direction dir, unsigned long flags)
+{
+ enum dma_slave_buswidth dev_width;
+ struct udma_desc *d;
+ size_t tr_size;
+ struct cppi50_tr_req_type1 *tr_req;
+ unsigned int i;
+ unsigned int periods = buf_len / period_len;
+ u32 burst;
+
+ if (dir == DMA_DEV_TO_MEM) {
+ dev_width = uc->cfg.src_addr_width;
+ burst = uc->cfg.src_maxburst;
+ } else if (dir == DMA_MEM_TO_DEV) {
+ dev_width = uc->cfg.dst_addr_width;
+ burst = uc->cfg.dst_maxburst;
+ } else {
+ dev_err(uc->ud->dev, "%s: bad direction?\n", __func__);
+ return NULL;
+ }
+
+ if (!burst)
+ burst = 1;
+
+ /* Now allocate and setup the descriptor. */
+ tr_size = sizeof(struct cppi50_tr_req_type1);
+ d = udma_alloc_tr_desc(uc, tr_size, periods, dir);
+ if (!d)
+ return NULL;
+
+ tr_req = (struct cppi50_tr_req_type1 *)d->hwdesc[0].tr_req_base;
+ for (i = 0; i < periods; i++) {
+ tr_req[i].flags = CPPI50_TR_FLAGS_TYPE(1);
+ tr_req[i].addr = buf_addr + period_len * i;
+ tr_req[i].icnt0 = dev_width;
+ tr_req[i].icnt1 = period_len / dev_width;
+ tr_req[i].dim1 = dev_width;
+
+ if (!(flags & DMA_PREP_INTERRUPT))
+ tr_req[i].flags |= CPPI50_TR_FLAGS_SUPR_EVT;
+ }
+
+ return d;
+}
+
+static struct udma_desc *udma_prep_dma_cyclic_pkt(
+ struct udma_chan *uc, dma_addr_t buf_addr, size_t buf_len,
+ size_t period_len, enum dma_transfer_direction dir, unsigned long flags)
+{
+ struct udma_desc *d;
+ u32 ring_id;
+ int i;
+ int periods = buf_len / period_len;
+
+ if (periods > 15)
+ return NULL;
+
+ if (period_len > 0x3FFFFF)
+ return NULL;
+
+ d = kzalloc(sizeof(*d) + periods * sizeof(d->hwdesc[0]), GFP_ATOMIC);
+ if (!d)
+ return NULL;
+
+ d->hwdesc_count = periods;
+
+ /* TODO: re-check this... */
+ if (dir == DMA_DEV_TO_MEM)
+ ring_id = k3_nav_ringacc_get_ring_id(uc->rchan->r_ring);
+ else
+ ring_id = k3_nav_ringacc_get_ring_id(uc->tchan->tc_ring);
+
+ for (i = 0; i < periods; i++) {
+ struct udma_hwdesc *hwdesc = &d->hwdesc[i];
+ dma_addr_t period_addr = buf_addr + (period_len * i);
+ struct knav_udmap_host_desc_t *h_desc;
+
+ hwdesc->cppi5_desc_vaddr = dma_pool_zalloc(uc->hdesc_pool,
+ GFP_ATOMIC,
+ &hwdesc->cppi5_desc_paddr);
+ if (!hwdesc->cppi5_desc_vaddr) {
+ dev_err(uc->ud->dev,
+ "descriptor%d allocation failed\n", i);
+
+ udma_free_hwdesc(&d->vd);
+ kfree(d);
+ return NULL;
+ }
+
+ hwdesc->cppi5_desc_size = uc->hdesc_size;
+ h_desc = hwdesc->cppi5_desc_vaddr;
+
+ knav_udmap_hdesc_init(h_desc, 0, 0);
+ knav_udmap_hdesc_set_pktlen(h_desc, period_len);
+
+ /* Flow and Packed ID ??? */
+ knav_udmap_hdesc_set_pktids(&h_desc->hdr, uc->id, 0x3fff);
+ knav_udmap_desc_set_retpolicy(&h_desc->hdr, 0, ring_id);
+
+ /* attach each period to a new descriptor */
+ knav_udmap_hdesc_attach_buf(h_desc,
+ period_addr, period_len,
+ period_addr, period_len);
+ }
+
+ return d;
+}
+
+static struct dma_async_tx_descriptor *udma_prep_dma_cyclic(
+ struct dma_chan *chan, dma_addr_t buf_addr, size_t buf_len,
+ size_t period_len, enum dma_transfer_direction dir, unsigned long flags)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ enum dma_slave_buswidth dev_width;
+ struct udma_desc *d;
+ u8 elsize;
+ u32 burst;
+
+ if (dir != uc->dir) {
+ dev_err(chan->device->dev,
+ "%s: chan%d is for %s, not supporting %s\n",
+ __func__, uc->id, udma_get_dir_text(uc->dir),
+ udma_get_dir_text(dir));
+ return NULL;
+ }
+
+ uc->cyclic = true;
+
+ if (dir == DMA_DEV_TO_MEM) {
+ dev_width = uc->cfg.src_addr_width;
+ burst = uc->cfg.src_maxburst;
+ } else if (dir == DMA_MEM_TO_DEV) {
+ dev_width = uc->cfg.dst_addr_width;
+ burst = uc->cfg.dst_maxburst;
+ } else {
+ dev_err(uc->ud->dev, "%s: bad direction?\n", __func__);
+ return NULL;
+ }
+
+ if (!burst)
+ burst = 1;
+
+ /* Bus width translates to the element size (ES) */
+ switch (dev_width) {
+ case DMA_SLAVE_BUSWIDTH_1_BYTE:
+ elsize = UDMA_ELSIZE_8;
+ break;
+ case DMA_SLAVE_BUSWIDTH_2_BYTES:
+ elsize = UDMA_ELSIZE_16;
+ break;
+ case DMA_SLAVE_BUSWIDTH_3_BYTES:
+ elsize = UDMA_ELSIZE_24;
+ break;
+ case DMA_SLAVE_BUSWIDTH_4_BYTES:
+ elsize = UDMA_ELSIZE_32;
+ break;
+ case DMA_SLAVE_BUSWIDTH_8_BYTES:
+ elsize = UDMA_ELSIZE_64;
+ break;
+ default: /* not reached */
+ return NULL;
+ }
+
+ if (uc->pkt_mode)
+ d = udma_prep_dma_cyclic_pkt(uc, buf_addr, buf_len, period_len,
+ dir, flags);
+ else
+ d = udma_prep_dma_cyclic_tr(uc, buf_addr, buf_len, period_len,
+ dir, flags);
+
+ if (!d)
+ return NULL;
+
+ d->sglen = buf_len / period_len;
+
+ d->dir = dir;
+ d->residue = buf_len;
+
+ /* static TR for remote PDMA */
+ if (udma_configure_statictr(uc, d, elsize, burst)) {
+ dev_err(uc->ud->dev,
+ "%s: StaticTR Z is limted to maximum 4095 (%u)\n",
+ __func__, d->static_tr.bstcnt);
+
+ udma_free_hwdesc(&d->vd);
+ kfree(d);
+ return NULL;
+ }
+
+ if (uc->metadata_size)
+ d->vd.tx.metadata_ops = &metadata_ops;
+
+ return vchan_tx_prep(&uc->vc, &d->vd, flags);
+}
+
+static struct dma_async_tx_descriptor *udma_prep_dma_memcpy(
+ struct dma_chan *chan, dma_addr_t dest, dma_addr_t src,
+ size_t len, unsigned long tx_flags)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ struct udma_desc *d;
+ struct cppi50_tr_req_type15 *tr_req;
+ int num_tr;
+ size_t tr_size = sizeof(struct cppi50_tr_req_type15);
+ u16 tr0_cnt0, tr0_cnt1, tr1_cnt0;
+
+ if (uc->dir != DMA_MEM_TO_MEM) {
+ dev_err(chan->device->dev,
+ "%s: chan%d is for %s, not supporting %s\n",
+ __func__, uc->id, udma_get_dir_text(uc->dir),
+ udma_get_dir_text(DMA_MEM_TO_MEM));
+ return NULL;
+ }
+
+ if (len < SZ_64K) {
+ num_tr = 1;
+ tr0_cnt0 = len;
+ tr0_cnt1 = 1;
+ } else {
+ unsigned long align_to = __ffs(src | dest);
+
+ if (align_to > 3)
+ align_to = 3;
+ /*
+ * Keep simple: tr0: SZ_64K-alignment blocks,
+ * tr1: the remaining
+ */
+ num_tr = 2;
+ tr0_cnt0 = (SZ_64K - BIT(align_to));
+ if (len / tr0_cnt0 >= SZ_64K) {
+ dev_err(uc->ud->dev, "size %zu is not supported\n",
+ len);
+ return NULL;
+ }
+
+ tr0_cnt1 = len / tr0_cnt0;
+ tr1_cnt0 = len % tr0_cnt0;
+ }
+
+ d = udma_alloc_tr_desc(uc, tr_size, num_tr, DMA_MEM_TO_MEM);
+ if (!d)
+ return NULL;
+
+ d->dir = DMA_MEM_TO_MEM;
+ d->desc_idx = 0;
+ d->tr_idx = 0;
+ d->residue = len;
+
+ tr_req = (struct cppi50_tr_req_type15 *)d->hwdesc[0].tr_req_base;
+
+ tr_req[0].flags = CPPI50_TR_FLAGS_TYPE(15);
+ tr_req[0].addr = src;
+ tr_req[0].icnt0 = tr0_cnt0;
+ tr_req[0].icnt1 = tr0_cnt1;
+ tr_req[0].icnt2 = 1;
+ tr_req[0].icnt3 = 1;
+ tr_req[0].dim1 = tr0_cnt0;
+
+ tr_req[0].daddr = dest;
+ tr_req[0].dicnt0 = tr0_cnt0;
+ tr_req[0].dicnt1 = tr0_cnt1;
+ tr_req[0].dicnt2 = 1;
+ tr_req[0].dicnt3 = 1;
+ tr_req[0].ddim1 = tr0_cnt0;
+
+ tr_req[0].flags |= CPPI50_TR_FLAGS_WAIT;
+
+ tr_req[0].flags |= CPPI50_TR_FLAGS_SUPR_EVT;
+
+ if (num_tr == 2) {
+ tr_req[1].flags = CPPI50_TR_FLAGS_TYPE(15);
+ tr_req[1].addr = src + tr0_cnt1 * tr0_cnt0;
+ tr_req[1].icnt0 = tr1_cnt0;
+ tr_req[1].icnt1 = 1;
+ tr_req[1].icnt2 = 1;
+ tr_req[1].icnt3 = 1;
+
+ tr_req[1].daddr = dest + tr0_cnt1 * tr0_cnt0;
+ tr_req[1].dicnt0 = tr1_cnt0;
+ tr_req[1].dicnt1 = 1;
+ tr_req[1].dicnt2 = 1;
+ tr_req[1].dicnt3 = 1;
+
+ tr_req[1].flags |= CPPI50_TR_FLAGS_WAIT;
+
+ tr_req[1].flags |= CPPI50_TR_FLAGS_SUPR_EVT;
+ }
+
+ tr_req[num_tr - 1].flags |= CPPI50_TR_FLAGS_EOP;
+
+ if (uc->metadata_size)
+ d->vd.tx.metadata_ops = &metadata_ops;
+
+ return vchan_tx_prep(&uc->vc, &d->vd, tx_flags);
+}
+
+static void udma_issue_pending(struct dma_chan *chan)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ unsigned long flags;
+
+ spin_lock_irqsave(&uc->vc.lock, flags);
+
+ /* If we have something pending and no active descriptor, then */
+ if (vchan_issue_pending(&uc->vc) && !uc->desc) {
+ /*
+ * start a descriptor if the channel is NOT [marked as
+ * terminating _and_ it is still running (teardown has not
+ * completed yet)].
+ */
+ if (!(uc->state == UDMA_CHAN_IS_TERMINATING &&
+ udma_is_chan_running(uc)))
+ udma_start(uc);
+ }
+
+ spin_unlock_irqrestore(&uc->vc.lock, flags);
+}
+
+/* Not much yet */
+static enum dma_status udma_tx_status(struct dma_chan *chan,
+ dma_cookie_t cookie,
+ struct dma_tx_state *txstate)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ enum dma_status ret;
+
+ ret = dma_cookie_status(chan, cookie, txstate);
+
+ if (!udma_is_chan_running(uc))
+ ret = DMA_COMPLETE;
+
+ if (ret == DMA_COMPLETE || !txstate)
+ return ret;
+
+ if (uc->desc && uc->desc->vd.tx.cookie == cookie) {
+ u32 pdma_bcnt = 0;
+ u32 bcnt = 0;
+ u32 pcnt = 0;
+ u32 residue = uc->desc->residue;
+ u32 delay = 0;
+
+ if (uc->desc->dir == DMA_MEM_TO_DEV) {
+ bcnt = udma_tchanrt_read(uc->tchan,
+ UDMA_TCHAN_RT_BCNT_REG);
+ pdma_bcnt = udma_tchanrt_read(uc->tchan,
+ UDMA_TCHAN_RT_PEER_BCNT_REG);
+ pcnt = udma_tchanrt_read(uc->tchan,
+ UDMA_TCHAN_RT_PCNT_REG);
+
+ if (bcnt > pdma_bcnt)
+ delay = bcnt - pdma_bcnt;
+ } else if (uc->desc->dir == DMA_DEV_TO_MEM) {
+ bcnt = udma_rchanrt_read(uc->rchan,
+ UDMA_RCHAN_RT_SBCNT_REG);
+ pdma_bcnt = udma_rchanrt_read(uc->rchan,
+ UDMA_RCHAN_RT_PEER_BCNT_REG);
+ pcnt = udma_rchanrt_read(uc->rchan,
+ UDMA_RCHAN_RT_PCNT_REG);
+
+ if (pdma_bcnt > bcnt)
+ delay = pdma_bcnt - bcnt;
+ } else {
+ u32 sbcnt;
+
+ sbcnt = udma_tchanrt_read(uc->tchan,
+ UDMA_TCHAN_RT_SBCNT_REG);
+ bcnt = udma_tchanrt_read(uc->tchan,
+ UDMA_TCHAN_RT_PEER_BCNT_REG);
+ pcnt = udma_tchanrt_read(uc->tchan,
+ UDMA_TCHAN_RT_PCNT_REG);
+
+ if (sbcnt > bcnt)
+ delay = sbcnt - bcnt;
+ }
+
+ residue -= ((bcnt - uc->bcnt) % uc->desc->residue);
+ dma_set_residue(txstate, residue);
+ dma_set_cached(txstate, delay);
+
+ } else {
+ ret = DMA_COMPLETE;
+ }
+
+ return ret;
+}
+
+static int udma_pause(struct dma_chan *chan)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+
+ if (!uc->desc)
+ return -EINVAL;
+
+ /* pause the channel */
+ switch (uc->desc->dir) {
+ case DMA_DEV_TO_MEM:
+ udma_rchanrt_update_bits(uc->rchan,
+ UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_PAUSE,
+ UDMA_PEER_RT_EN_PAUSE);
+ break;
+ case DMA_MEM_TO_DEV:
+ udma_tchanrt_update_bits(uc->tchan,
+ UDMA_TCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_PAUSE,
+ UDMA_PEER_RT_EN_PAUSE);
+ break;
+ case DMA_MEM_TO_MEM:
+ udma_tchanrt_update_bits(uc->tchan, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_PAUSE,
+ UDMA_CHAN_RT_CTL_PAUSE);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int udma_resume(struct dma_chan *chan)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+
+ if (!uc->desc)
+ return -EINVAL;
+
+ /* resume the channel */
+ switch (uc->desc->dir) {
+ case DMA_DEV_TO_MEM:
+ udma_rchanrt_update_bits(uc->rchan,
+ UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_PAUSE, 0);
+
+ break;
+ case DMA_MEM_TO_DEV:
+ udma_tchanrt_update_bits(uc->tchan,
+ UDMA_TCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_PAUSE, 0);
+ break;
+ case DMA_MEM_TO_MEM:
+ udma_tchanrt_update_bits(uc->tchan, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_PAUSE, 0);
+ break;
+ default:
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int udma_terminate_all(struct dma_chan *chan)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ unsigned long flags;
+ LIST_HEAD(head);
+
+ spin_lock_irqsave(&uc->vc.lock, flags);
+
+ if (udma_is_chan_running(uc))
+ udma_stop(uc);
+
+ if (uc->desc) {
+ uc->terminated_desc = uc->desc;
+ uc->desc = NULL;
+ uc->terminated_desc->terminated = true;
+ }
+
+ uc->paused = false;
+
+ vchan_get_all_descriptors(&uc->vc, &head);
+ spin_unlock_irqrestore(&uc->vc.lock, flags);
+ vchan_dma_desc_free_list(&uc->vc, &head);
+
+ return 0;
+}
+
+static void udma_synchronize(struct dma_chan *chan)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ unsigned long timeout = msecs_to_jiffies(1000);
+
+ vchan_synchronize(&uc->vc);
+
+ if (uc->state == UDMA_CHAN_IS_TERMINATING) {
+ timeout = wait_for_completion_timeout(&uc->teardown_completed,
+ timeout);
+ if (!timeout) {
+ dev_warn(uc->ud->dev, "chan%d teardown timeout!\n",
+ uc->id);
+ udma_dump_chan_stdata(uc);
+ udma_reset_chan(uc, true);
+ }
+ }
+
+ udma_reset_chan(uc, false);
+ if (udma_is_chan_running(uc))
+ dev_warn(uc->ud->dev, "chan%d refused to stop!\n", uc->id);
+
+ udma_reset_rings(uc);
+}
+
+static void udma_desc_pre_callback(struct virt_dma_chan *vc,
+ struct virt_dma_desc *vd,
+ struct dmaengine_result *result)
+{
+ struct udma_chan *uc = to_udma_chan(&vc->chan);
+ struct udma_desc *d;
+
+ if (!vd)
+ return;
+
+ d = to_udma_desc(&vd->tx);
+
+ if (d->metadata_size)
+ udma_fetch_epib(uc, d);
+
+ /* TODO: peek into the desc to know the real length */
+ if (d->rx_sg_wa.in_use) {
+ void *src = sg_virt(&d->rx_sg_wa.single_sg);
+
+ dma_sync_sg_for_cpu(uc->ud->dev, &d->rx_sg_wa.single_sg, 1,
+ DMA_FROM_DEVICE);
+ /* Ensure that reads are not moved before this point */
+ rmb();
+
+ sg_copy_from_buffer(d->rx_sg_wa.sgl, d->rx_sg_wa.sglen, src,
+ d->rx_sg_wa.total_len);
+ }
+
+ /* Provide residue information for the client */
+ if (result) {
+ void *desc_vaddr = udma_curr_cppi5_desc_vaddr(d, d->desc_idx);
+
+ if (knav_udmap_desc_get_type(desc_vaddr) ==
+ KNAV_UDMAP_INFO0_DESC_TYPE_VAL_HOST) {
+ result->residue = knav_udmap_hdesc_get_pktlen(desc_vaddr);
+ if (result->residue == d->residue)
+ result->result = DMA_TRANS_NOERROR;
+ else
+ result->result = DMA_TRANS_ABORTED;
+ } else {
+ result->residue = d->residue;
+ result->result = DMA_TRANS_NOERROR;
+ }
+ }
+}
+
+/*
+ * This tasklet handles the completion of a DMA descriptor by
+ * calling its callback and freeing it.
+ */
+static void udma_vchan_complete(unsigned long arg)
+{
+ struct virt_dma_chan *vc = (struct virt_dma_chan *)arg;
+ struct virt_dma_desc *vd, *_vd;
+ struct dmaengine_desc_callback cb;
+ LIST_HEAD(head);
+
+ spin_lock_irq(&vc->lock);
+ list_splice_tail_init(&vc->desc_completed, &head);
+ vd = vc->cyclic;
+ if (vd) {
+ vc->cyclic = NULL;
+ dmaengine_desc_get_callback(&vd->tx, &cb);
+ } else {
+ memset(&cb, 0, sizeof(cb));
+ }
+ spin_unlock_irq(&vc->lock);
+
+ udma_desc_pre_callback(vc, vd, NULL);
+ dmaengine_desc_callback_invoke(&cb, NULL);
+
+ list_for_each_entry_safe(vd, _vd, &head, node) {
+ struct dmaengine_result result;
+
+ dmaengine_desc_get_callback(&vd->tx, &cb);
+
+ list_del(&vd->node);
+
+ udma_desc_pre_callback(vc, vd, &result);
+ dmaengine_desc_callback_invoke(&cb, &result);
+
+ vchan_vdesc_fini(vd);
+ }
+}
+
+static void udma_free_chan_resources(struct dma_chan *chan)
+{
+ struct udma_chan *uc = to_udma_chan(chan);
+ struct udma_dev *ud = to_udma_dev(chan->device);
+ struct udma_tisci_rm *tisci_rm = &ud->tisci_rm;
+
+ udma_terminate_all(chan);
+
+ if (uc->irq_num_ring > 0) {
+ free_irq(uc->irq_num_ring, uc);
+
+ ti_sci_inta_unregister_event(ud->dev, uc->irq_ra_tisci,
+ uc->irq_ra_idx, uc->irq_num_ring);
+ uc->irq_num_ring = 0;
+ }
+ if (uc->irq_num_udma > 0) {
+ free_irq(uc->irq_num_udma, uc);
+
+ ti_sci_inta_unregister_event(ud->dev, tisci_rm->tisci_dev_id,
+ uc->irq_udma_idx,
+ uc->irq_num_udma);
+ uc->irq_num_udma = 0;
+ }
+
+ /* Release PSI-L pairing */
+ if (uc->psil_paired) {
+ navss_psil_unpair(ud, uc->src_thread, uc->dst_thread);
+ uc->psil_paired = false;
+ }
+
+ vchan_free_chan_resources(&uc->vc);
+ tasklet_kill(&uc->vc.task);
+
+ pm_runtime_put(ud->ddev.dev);
+
+ udma_free_tx_resources(uc);
+ udma_free_rx_resources(uc);
+
+ uc->slave_thread_id = -1;
+ uc->dir = DMA_MEM_TO_MEM;
+
+ if (uc->use_dma_pool) {
+ dma_pool_destroy(uc->hdesc_pool);
+ uc->use_dma_pool = false;
+ }
+}
+
+static struct platform_driver udma_driver;
+
+static bool udma_dma_filter_fn(struct dma_chan *chan, void *param)
+{
+ u32 *args;
+ struct udma_chan *uc;
+ struct udma_dev *ud;
+ struct device_node *chconf_node, *slave_node;
+ char prop[50];
+ u32 val;
+
+ if (chan->device->dev->driver != &udma_driver.driver)
+ return false;
+
+ uc = to_udma_chan(chan);
+ ud = uc->ud;
+ args = param;
+
+ if (args[2] == UDMA_DIR_TX) {
+ uc->dir = DMA_MEM_TO_DEV;
+ } else if (args[2] == UDMA_DIR_RX) {
+ uc->dir = DMA_DEV_TO_MEM;
+ } else {
+ dev_err(ud->dev, "Invalid direction (%u)\n", args[2]);
+ return false;
+ }
+
+ slave_node = of_find_node_by_phandle(args[0]);
+ if (!slave_node) {
+ dev_err(ud->dev, "Slave node is missing\n");
+ return false;
+ }
+
+ snprintf(prop, sizeof(prop), "ti,psil-config%u", args[1]);
+ chconf_node = of_find_node_by_name(slave_node, prop);
+ if (!chconf_node) {
+ dev_err(ud->dev, "Channel configuration node is missing\n");
+ of_node_put(slave_node);
+ return false;
+ }
+
+ if (!of_property_read_u32(chconf_node, "linux,udma-mode", &val)) {
+ if (val == UDMA_PKT_MODE)
+ uc->pkt_mode = true;
+ }
+
+ if (!of_property_read_u32(chconf_node, "statictr-type", &val))
+ uc->static_tr_type = val;
+
+ if (!of_property_read_u32(chconf_node, "ti,channel-tpl", &val))
+ uc->channel_tpl = val;
+
+ uc->needs_epib = of_property_read_bool(chconf_node, "ti,needs-epib");
+ if (!of_property_read_u32(chconf_node, "ti,psd-size", &val))
+ uc->psd_size = val;
+ uc->metadata_size = (uc->needs_epib ? 16 : 0) + uc->psd_size;
+
+ uc->desc_align = 64;
+ if (uc->desc_align < dma_get_cache_alignment())
+ uc->desc_align = dma_get_cache_alignment();
+
+ if (uc->pkt_mode)
+ uc->hdesc_size = ALIGN(sizeof(struct knav_udmap_host_desc_t) +
+ uc->metadata_size, uc->desc_align);
+
+ of_node_put(chconf_node);
+
+ if (of_property_read_u32(slave_node, "ti,psil-base", &val)) {
+ dev_err(ud->dev, "ti,psil-base is missing\n");
+ of_node_put(slave_node);
+ return false;
+ }
+
+ uc->slave_thread_id = val + args[1];
+
+ of_node_put(slave_node);
+
+ dev_dbg(ud->dev, "%s: Slave %s thread%d will be handled by vchan %d\n",
+ __func__, udma_get_dir_text(uc->dir), uc->slave_thread_id,
+ uc->id);
+
+ return true;
+}
+
+static struct dma_chan *udma_of_xlate(struct of_phandle_args *dma_spec,
+ struct of_dma *ofdma)
+{
+ struct udma_dev *ud = ofdma->of_dma_data;
+ struct dma_chan *chan;
+
+ if (dma_spec->args_count != 3)
+ return NULL;
+
+ chan = dmadev_get_slave_channel(&ud->ddev, udma_dma_filter_fn,
+ &dma_spec->args[0]);
+ if (!chan) {
+ dev_err(ud->dev, "get channel fail in %s.\n", __func__);
+ return ERR_PTR(-EINVAL);
+ }
+
+ return chan;
+}
+
+struct udma_match_data am654_main_data = {
+ .tpl_levels = 2,
+ .level_start_idx = {
+ [0] = 8, /* Normal channels */
+ [1] = 0, /* High Throughput channels */
+ },
+};
+
+struct udma_match_data am654_mcu_data = {
+ .tpl_levels = 2,
+ .level_start_idx = {
+ [0] = 2, /* Normal channels */
+ [1] = 0, /* High Throughput channels */
+ },
+};
+
+static const struct of_device_id udma_of_match[] = {
+ { .compatible = "ti,am654-navss-main-udmap", .data = &am654_main_data, },
+ { .compatible = "ti,am654-navss-mcu-udmap", .data = &am654_mcu_data, },
+ {},
+};
+MODULE_DEVICE_TABLE(of, udma_of_match);
+
+static int udma_get_mmrs(struct platform_device *pdev, struct udma_dev *ud)
+{
+ struct resource *res;
+ int i;
+
+ for (i = 0; i < MMR_LAST; i++) {
+ res = platform_get_resource_byname(pdev, IORESOURCE_MEM,
+ mmr_names[i]);
+ ud->mmrs[i] = devm_ioremap_resource(&pdev->dev, res);
+ if (IS_ERR(ud->mmrs[i]))
+ return PTR_ERR(ud->mmrs[i]);
+ }
+
+ return 0;
+}
+
+static int udma_setup_resources(struct udma_dev *ud)
+{
+ struct device *dev = ud->dev;
+ int ch_count, i;
+ u32 cap2, cap3;
+ struct ti_sci_resource_desc *rm_desc;
+ struct ti_sci_resource *rm_res;
+ struct udma_tisci_rm *tisci_rm = &ud->tisci_rm;
+ char *range_names[] = { "ti,sci-rm-range-tchan",
+ "ti,sci-rm-range-rchan",
+ "ti,sci-rm-range-rflow" };
+
+ cap2 = udma_read(ud->mmrs[MMR_GCFG], 0x28);
+ cap3 = udma_read(ud->mmrs[MMR_GCFG], 0x2c);
+
+ ud->rflow_cnt = cap3 & 0x3fff;
+ ud->tchan_cnt = cap2 & 0x1ff;
+ ud->echan_cnt = (cap2 >> 9) & 0x1ff;
+ ud->rchan_cnt = (cap2 >> 18) & 0x1ff;
+ ch_count = ud->tchan_cnt + ud->rchan_cnt;
+
+ ud->tchan_map = devm_kmalloc_array(dev, BITS_TO_LONGS(ud->tchan_cnt),
+ sizeof(unsigned long), GFP_KERNEL);
+ ud->tchans = devm_kcalloc(dev, ud->tchan_cnt, sizeof(*ud->tchans),
+ GFP_KERNEL);
+ ud->rchan_map = devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rchan_cnt),
+ sizeof(unsigned long), GFP_KERNEL);
+ ud->rchans = devm_kcalloc(dev, ud->rchan_cnt, sizeof(*ud->rchans),
+ GFP_KERNEL);
+ ud->rflow_map = devm_kmalloc_array(dev, BITS_TO_LONGS(ud->rflow_cnt),
+ sizeof(unsigned long), GFP_KERNEL);
+ ud->rflow_map_reserved = devm_kcalloc(dev, BITS_TO_LONGS(ud->rflow_cnt),
+ sizeof(unsigned long),
+ GFP_KERNEL);
+ ud->rflows = devm_kcalloc(dev, ud->rflow_cnt, sizeof(*ud->rflows),
+ GFP_KERNEL);
+
+ if (!ud->tchan_map || !ud->rchan_map || !ud->rflow_map ||
+ !ud->rflow_map_reserved || !ud->tchans || !ud->rchans ||
+ !ud->rflows)
+ return -ENOMEM;
+
+ /*
+ * RX flows with the same Ids as RX channels are reserved to be used
+ * as default flows if remote HW can't generate flow_ids. Those
+ * RX flows can be requested only explicitly by id.
+ */
+ bitmap_set(ud->rflow_map_reserved, 0, ud->rchan_cnt);
+
+ /* Get resource ranges from tisci */
+ for (i = 0; i < RM_RANGE_LAST; i++)
+ tisci_rm->rm_ranges[i] = devm_ti_sci_get_of_resource(
+ tisci_rm->tisci, dev,
+ range_names[i]);
+
+ /* tchan ranges */
+ rm_res = tisci_rm->rm_ranges[RM_RANGE_TCHAN];
+ if (IS_ERR(rm_res)) {
+ bitmap_zero(ud->tchan_map, ud->tchan_cnt);
+ } else {
+ bitmap_fill(ud->tchan_map, ud->tchan_cnt);
+ for (i = 0; i < rm_res->sets; i++) {
+ rm_desc = &rm_res->desc[i];
+ bitmap_clear(ud->tchan_map, rm_desc->start,
+ rm_desc->num);
+ }
+ }
+
+ /* rchan and matching default flow ranges */
+ rm_res = tisci_rm->rm_ranges[RM_RANGE_RCHAN];
+ if (IS_ERR(rm_res)) {
+ bitmap_zero(ud->rchan_map, ud->rchan_cnt);
+ bitmap_zero(ud->rflow_map, ud->rchan_cnt);
+ } else {
+ bitmap_fill(ud->rchan_map, ud->rchan_cnt);
+ bitmap_fill(ud->rflow_map, ud->rchan_cnt);
+ for (i = 0; i < rm_res->sets; i++) {
+ rm_desc = &rm_res->desc[i];
+ bitmap_clear(ud->rchan_map, rm_desc->start,
+ rm_desc->num);
+ bitmap_clear(ud->rflow_map, rm_desc->start,
+ rm_desc->num);
+ }
+ }
+
+ /* GP rflow ranges */
+ rm_res = tisci_rm->rm_ranges[RM_RANGE_RFLOW];
+ if (IS_ERR(rm_res)) {
+ bitmap_clear(ud->rflow_map, ud->rchan_cnt,
+ ud->rflow_cnt - ud->rchan_cnt);
+ } else {
+ bitmap_set(ud->rflow_map, ud->rchan_cnt,
+ ud->rflow_cnt - ud->rchan_cnt);
+ for (i = 0; i < rm_res->sets; i++) {
+ rm_desc = &rm_res->desc[i];
+ bitmap_clear(ud->rflow_map, rm_desc->start,
+ rm_desc->num);
+ }
+ }
+
+ /*
+ * HACK: tchan0, rchan0,1 and rflow0,1 on main_navss is dedicated to
+ * sysfw.
+ * Only UDMAP on main_navss have echan, use it as a hint for now.
+ */
+ if (ud->echan_cnt) {
+ set_bit(0, ud->tchan_map);
+ set_bit(0, ud->rchan_map);
+ set_bit(1, ud->rchan_map);
+ set_bit(0, ud->rflow_map);
+ set_bit(1, ud->rflow_map);
+ }
+
+ ch_count -= bitmap_weight(ud->tchan_map, ud->tchan_cnt);
+ ch_count -= bitmap_weight(ud->rchan_map, ud->rchan_cnt);
+ if (!ch_count)
+ return -ENODEV;
+
+ ud->channels = devm_kcalloc(dev, ch_count, sizeof(*ud->channels),
+ GFP_KERNEL);
+ if (!ud->channels)
+ return -ENOMEM;
+
+ dev_info(dev,
+ "Channels: %d (tchan: %u, echan: %u, rchan: %u, rflow: %u)\n",
+ ch_count, ud->tchan_cnt, ud->echan_cnt, ud->rchan_cnt,
+ ud->rflow_cnt);
+
+ return ch_count;
+}
+
+#define TI_UDMAC_BUSWIDTHS (BIT(DMA_SLAVE_BUSWIDTH_1_BYTE) | \
+ BIT(DMA_SLAVE_BUSWIDTH_2_BYTES) | \
+ BIT(DMA_SLAVE_BUSWIDTH_3_BYTES) | \
+ BIT(DMA_SLAVE_BUSWIDTH_4_BYTES) | \
+ BIT(DMA_SLAVE_BUSWIDTH_8_BYTES))
+
+static int udma_probe(struct platform_device *pdev)
+{
+ struct device_node *parent_irq_node;
+ struct device_node *navss_node = pdev->dev.parent->of_node;
+ struct device *dev = &pdev->dev;
+ struct udma_dev *ud;
+ const struct of_device_id *match;
+ int i, ret;
+ int ch_count;
+
+ ret = dma_coerce_mask_and_coherent(dev, DMA_BIT_MASK(48));
+ if (ret)
+ dev_err(dev, "failed to set dma mask stuff\n");
+
+ ud = devm_kzalloc(dev, sizeof(*ud), GFP_KERNEL);
+ if (!ud)
+ return -ENOMEM;
+
+ ret = udma_get_mmrs(pdev, ud);
+ if (ret)
+ return ret;
+
+ ud->tisci_rm.tisci = ti_sci_get_by_phandle(dev->of_node, "ti,sci");
+ if (IS_ERR(ud->tisci_rm.tisci))
+ return PTR_ERR(ud->tisci_rm.tisci);
+
+ ret = of_property_read_u32(dev->of_node, "ti,sci-dev-id",
+ &ud->tisci_rm.tisci_dev_id);
+ if (ret) {
+ dev_err(dev, "ti,sci-dev-id read failure %d\n", ret);
+ return ret;
+ }
+
+ ret = of_property_read_u32(navss_node, "ti,sci-dev-id",
+ &ud->tisci_rm.tisci_navss_dev_id);
+ if (ret) {
+ dev_err(dev, "NAVSS ti,sci-dev-id read failure %d\n", ret);
+ return ret;
+ }
+
+ ud->tisci_rm.tisci_udmap_ops = &ud->tisci_rm.tisci->ops.rm_udmap_ops;
+ ud->tisci_rm.tisci_psil_ops = &ud->tisci_rm.tisci->ops.rm_psil_ops;
+
+ ud->ringacc = of_k3_nav_ringacc_get_by_phandle(dev->of_node,
+ "ti,ringacc");
+ if (IS_ERR(ud->ringacc))
+ return PTR_ERR(ud->ringacc);
+
+ parent_irq_node = of_irq_find_parent(dev->of_node);
+ if (!parent_irq_node) {
+ dev_err(dev, "Failed to get IRQ parent node\n");
+ return -ENODEV;
+ }
+
+ ud->irq_domain = irq_find_host(parent_irq_node);
+ if (!ud->irq_domain)
+ return -EPROBE_DEFER;
+
+ match = of_match_node(udma_of_match, dev->of_node);
+ if (!match) {
+ dev_err(dev, "No compatible match found\n");
+ return -ENODEV;
+ }
+ ud->match_data = match->data;
+
+ dma_cap_set(DMA_SLAVE, ud->ddev.cap_mask);
+ dma_cap_set(DMA_CYCLIC, ud->ddev.cap_mask);
+ dma_cap_set(DMA_MEMCPY, ud->ddev.cap_mask);
+
+ ud->ddev.device_alloc_chan_resources = udma_alloc_chan_resources;
+ ud->ddev.device_config = udma_slave_config;
+ ud->ddev.device_prep_slave_sg = udma_prep_slave_sg;
+ ud->ddev.device_prep_dma_cyclic = udma_prep_dma_cyclic;
+ ud->ddev.device_prep_dma_memcpy = udma_prep_dma_memcpy;
+ ud->ddev.device_issue_pending = udma_issue_pending;
+ ud->ddev.device_tx_status = udma_tx_status;
+ ud->ddev.device_pause = udma_pause;
+ ud->ddev.device_resume = udma_resume;
+ ud->ddev.device_terminate_all = udma_terminate_all;
+ ud->ddev.device_synchronize = udma_synchronize;
+
+ ud->ddev.device_free_chan_resources = udma_free_chan_resources;
+ ud->ddev.src_addr_widths = TI_UDMAC_BUSWIDTHS;
+ ud->ddev.dst_addr_widths = TI_UDMAC_BUSWIDTHS;
+ ud->ddev.directions = BIT(DMA_DEV_TO_MEM) | BIT(DMA_MEM_TO_DEV) |
+ BIT(DMA_MEM_TO_MEM);
+ ud->ddev.residue_granularity = DMA_RESIDUE_GRANULARITY_BURST;
+ ud->ddev.copy_align = DMAENGINE_ALIGN_8_BYTES;
+ ud->ddev.desc_metadata_modes = DESC_METADATA_CLIENT |
+ DESC_METADATA_ENGINE;
+ ud->ddev.dev = dev;
+ ud->dev = dev;
+
+ INIT_LIST_HEAD(&ud->ddev.channels);
+ INIT_LIST_HEAD(&ud->desc_to_purge);
+
+ ret = of_property_read_u32(dev->of_node, "ti,psil-base",
+ &ud->psil_base);
+ if (ret) {
+ dev_info(dev, "Missing ti,psil-base property, using %d.\n",
+ ret);
+ return ret;
+ }
+
+ ch_count = udma_setup_resources(ud);
+ if (ch_count <= 0)
+ return ch_count;
+
+ spin_lock_init(&ud->lock);
+ INIT_WORK(&ud->purge_work, udma_purge_desc_work);
+
+ for (i = 0; i < ud->tchan_cnt; i++) {
+ struct udma_tchan *tchan = &ud->tchans[i];
+
+ tchan->id = i;
+ tchan->reg_rt = ud->mmrs[MMR_TCHANRT] + UDMA_CH_1000(i);
+ }
+
+ for (i = 0; i < ud->rchan_cnt; i++) {
+ struct udma_rchan *rchan = &ud->rchans[i];
+
+ rchan->id = i;
+ rchan->reg_rt = ud->mmrs[MMR_RCHANRT] + UDMA_CH_1000(i);
+ }
+
+ for (i = 0; i < ud->rflow_cnt; i++) {
+ struct udma_rflow *rflow = &ud->rflows[i];
+
+ rflow->id = i;
+ }
+
+ for (i = 0; i < ch_count; i++) {
+ struct udma_chan *uc = &ud->channels[i];
+
+ uc->ud = ud;
+ uc->vc.desc_free = udma_desc_free;
+ uc->id = i;
+ uc->slave_thread_id = -1;
+ uc->tchan = NULL;
+ uc->rchan = NULL;
+ uc->dir = DMA_MEM_TO_MEM;
+ uc->name = devm_kasprintf(dev, GFP_KERNEL, "UDMA chan%d", i);
+
+ vchan_init(&uc->vc, &ud->ddev);
+ /* Use custom vchan completion handling */
+ tasklet_init(&uc->vc.task, udma_vchan_complete,
+ (unsigned long)&uc->vc);
+ init_completion(&uc->teardown_completed);
+ }
+
+ ret = dma_async_device_register(&ud->ddev);
+ if (ret) {
+ dev_err(dev, "failed to register slave DMA engine: %d\n", ret);
+ return ret;
+ }
+
+ platform_set_drvdata(pdev, ud);
+
+ ret = of_dma_controller_register(dev->of_node, udma_of_xlate, ud);
+ if (ret) {
+ dev_err(dev, "failed to register of_dma controller\n");
+ dma_async_device_unregister(&ud->ddev);
+ }
+
+ return ret;
+}
+
+static int udma_remove(struct platform_device *pdev)
+{
+ struct udma_dev *ud = platform_get_drvdata(pdev);
+
+ of_dma_controller_free(pdev->dev.of_node);
+ dma_async_device_unregister(&ud->ddev);
+
+ /* Make sure that we did proper cleanup */
+ cancel_work_sync(&ud->purge_work);
+ udma_purge_desc_work(&ud->purge_work);
+
+ pm_runtime_put_sync(&pdev->dev);
+ pm_runtime_disable(&pdev->dev);
+
+ return 0;
+}
+
+static struct platform_driver udma_driver = {
+ .driver = {
+ .name = "ti-udma",
+ .of_match_table = udma_of_match,
+ },
+ .probe = udma_probe,
+ .remove = udma_remove,
+};
+
+module_platform_driver(udma_driver);
+
+MODULE_ALIAS("platform:ti-udma");
+MODULE_DESCRIPTION("TI K3 DMA driver for CPPI 5.0 compliant devices");
+MODULE_AUTHOR("Peter Ujfalusi <peter.ujfalusi@ti.com>");
+MODULE_LICENSE("GPL v2");
diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h
new file mode 100644
index 000000000000..787b9feefe5d
--- /dev/null
+++ b/drivers/dma/ti/k3-udma.h
@@ -0,0 +1,132 @@
+/* SPDX-License-Identifier: GPL-2.0 */
+/*
+ * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com
+ */
+
+#ifndef K3_UDMA_H_
+#define K3_UDMA_H_
+
+#define UDMA_PSIL_DST_THREAD_ID_OFFSET 0x8000
+
+/* Global registers */
+#define UDMA_REV_REG 0x0
+#define UDMA_PERF_CTL_REG 0x4
+#define UDMA_EMU_CTL_REG 0x8
+#define UDMA_PSIL_TO_REG 0x10
+#define UDMA_UTC_CTL_REG 0x1c
+#define UDMA_CAP_REG(i) (0x20 + (i * 4))
+#define UDMA_RX_FLOW_ID_FW_OES_REG 0x80
+#define UDMA_RX_FLOW_ID_FW_STATUS_REG 0x88
+
+/* TX chan RT regs */
+#define UDMA_TCHAN_RT_CTL_REG 0x0
+#define UDMA_TCHAN_RT_SWTRIG_REG 0x8
+#define UDMA_TCHAN_RT_STDATA_REG 0x80
+
+#define UDMA_TCHAN_RT_PEERn_REG(i) (0x200 + (i * 0x4))
+#define UDMA_TCHAN_RT_PEER_STATIC_TR_XY_REG \
+ UDMA_TCHAN_RT_PEERn_REG(0) /* PSI-L: 0x400 */
+#define UDMA_TCHAN_RT_PEER_STATIC_TR_Z_REG \
+ UDMA_TCHAN_RT_PEERn_REG(1) /* PSI-L: 0x401 */
+#define UDMA_TCHAN_RT_PEER_BCNT_REG \
+ UDMA_TCHAN_RT_PEERn_REG(4) /* PSI-L: 0x404 */
+#define UDMA_TCHAN_RT_PEER_RT_EN_REG \
+ UDMA_TCHAN_RT_PEERn_REG(8) /* PSI-L: 0x408 */
+
+#define UDMA_TCHAN_RT_PCNT_REG 0x400
+#define UDMA_TCHAN_RT_BCNT_REG 0x408
+#define UDMA_TCHAN_RT_SBCNT_REG 0x410
+
+/* RX chan RT regs */
+#define UDMA_RCHAN_RT_CTL_REG 0x0
+#define UDMA_RCHAN_RT_SWTRIG_REG 0x8
+#define UDMA_RCHAN_RT_STDATA_REG 0x80
+
+#define UDMA_RCHAN_RT_PEERn_REG(i) (0x200 + (i * 0x4))
+#define UDMA_RCHAN_RT_PEER_STATIC_TR_XY_REG \
+ UDMA_RCHAN_RT_PEERn_REG(0) /* PSI-L: 0x400 */
+#define UDMA_RCHAN_RT_PEER_STATIC_TR_Z_REG \
+ UDMA_RCHAN_RT_PEERn_REG(1) /* PSI-L: 0x401 */
+#define UDMA_RCHAN_RT_PEER_BCNT_REG \
+ UDMA_RCHAN_RT_PEERn_REG(4) /* PSI-L: 0x404 */
+#define UDMA_RCHAN_RT_PEER_RT_EN_REG \
+ UDMA_RCHAN_RT_PEERn_REG(8) /* PSI-L: 0x408 */
+
+#define UDMA_RCHAN_RT_PCNT_REG 0x400
+#define UDMA_RCHAN_RT_BCNT_REG 0x408
+#define UDMA_RCHAN_RT_SBCNT_REG 0x410
+
+/* UDMA_TCHAN_RT_CTL_REG/UDMA_RCHAN_RT_CTL_REG */
+#define UDMA_CHAN_RT_CTL_EN BIT(31)
+#define UDMA_CHAN_RT_CTL_TDOWN BIT(30)
+#define UDMA_CHAN_RT_CTL_PAUSE BIT(29)
+#define UDMA_CHAN_RT_CTL_FTDOWN BIT(28)
+#define UDMA_CHAN_RT_CTL_ERROR BIT(0)
+
+/* UDMA_TCHAN_RT_PEER_RT_EN_REG/UDMA_RCHAN_RT_PEER_RT_EN_REG (PSI-L: 0x408) */
+#define UDMA_PEER_RT_EN_ENABLE BIT(31)
+#define UDMA_PEER_RT_EN_TEARDOWN BIT(30)
+#define UDMA_PEER_RT_EN_PAUSE BIT(29)
+#define UDMA_PEER_RT_EN_FLUSH BIT(28)
+#define UDMA_PEER_RT_EN_IDLE BIT(1)
+
+/*
+ * UDMA_TCHAN_RT_PEER_STATIC_TR_XY_REG /
+ * UDMA_RCHAN_RT_PEER_STATIC_TR_XY_REG
+ */
+#define PDMA_STATIC_TR_X_MASK GENMASK(26, 24)
+#define PDMA_STATIC_TR_X_SHIFT (24)
+#define PDMA_STATIC_TR_Y_MASK GENMASK(11, 0)
+#define PDMA_STATIC_TR_Y_SHIFT (0)
+
+#define PDMA_STATIC_TR_Y(x) \
+ (((x) << PDMA_STATIC_TR_Y_SHIFT) & PDMA_STATIC_TR_Y_MASK)
+#define PDMA_STATIC_TR_X(x) \
+ (((x) << PDMA_STATIC_TR_X_SHIFT) & PDMA_STATIC_TR_X_MASK)
+
+/*
+ * UDMA_TCHAN_RT_PEER_STATIC_TR_Z_REG /
+ * UDMA_RCHAN_RT_PEER_STATIC_TR_Z_REG
+ */
+#define PDMA_STATIC_TR_Z_MASK GENMASK(11, 0)
+#define PDMA_STATIC_TR_Z_SHIFT (0)
+#define PDMA_STATIC_TR_Z(x) \
+ (((x) << PDMA_STATIC_TR_Z_SHIFT) & PDMA_STATIC_TR_Z_MASK)
+
+/* Private API to UDMA via k3-udma-private */
+#include <linux/of.h>
+#include <linux/dmaengine.h>
+#include <linux/soc/ti/ti_sci_protocol.h>
+
+struct udma_dev;
+struct udma_tchan;
+struct udma_rchan;
+struct udma_rflow;
+
+enum udma_rm_range {
+ RM_RANGE_TCHAN = 0,
+ RM_RANGE_RCHAN,
+ RM_RANGE_RFLOW,
+ RM_RANGE_LAST,
+};
+
+/* Channel Throughput Levels */
+enum udma_tp_level {
+ UDMA_TP_NORMAL = 0,
+ UDMA_TP_HIGH = 1,
+ UDMA_TP_LAST,
+};
+
+struct udma_tisci_rm {
+ const struct ti_sci_handle *tisci;
+ const struct ti_sci_rm_udmap_ops *tisci_udmap_ops;
+ u32 tisci_dev_id;
+
+ /* tisci information for PSI-L thread pairing/unpairing */
+ const struct ti_sci_rm_psil_ops *tisci_psil_ops;
+ u32 tisci_navss_dev_id;
+
+ struct ti_sci_resource *rm_ranges[RM_RANGE_LAST];
+};
+
+#endif /* K3_UDMA_H_ */
diff --git a/include/dt-bindings/dma/k3-udma.h b/include/dt-bindings/dma/k3-udma.h
new file mode 100644
index 000000000000..89ba6a9d4a8f
--- /dev/null
+++ b/include/dt-bindings/dma/k3-udma.h
@@ -0,0 +1,26 @@
+#ifndef __DT_TI_UDMA_H
+#define __DT_TI_UDMA_H
+
+#define UDMA_TR_MODE 0
+#define UDMA_PKT_MODE 1
+
+#define UDMA_DIR_TX 0
+#define UDMA_DIR_RX 1
+
+#define PSIL_STATIC_TR_NONE 0
+#define PSIL_STATIC_TR_XY 1
+#define PSIL_STATIC_TR_MCAN 2
+
+#define UDMA_PDMA_TR_XY(id) \
+ ti,psil-config##id { \
+ linux,udma-mode = <UDMA_TR_MODE>; \
+ statictr-type = <PSIL_STATIC_TR_XY>; \
+ }
+
+#define UDMA_PDMA_PKT_XY(id) \
+ ti,psil-config##id { \
+ linux,udma-mode = <UDMA_PKT_MODE>; \
+ statictr-type = <PSIL_STATIC_TR_XY>; \
+ }
+
+#endif /* __DT_TI_UDMA_H */
^ permalink raw reply related
* [RFC,5/5] dmaengine: ti: k3-udma: Add glue layer for non DMAengine users
From: Peter Ujfalusi @ 2018-09-24 13:00 UTC (permalink / raw)
To: vkoul, dan.j.williams; +Cc: dmaengine, grygorii.strashko
From: Grygorii Strashko <grygorii.strashko@ti.com>
Certain users can not use right now the DMAengine API due to missing
features in the core. Prime example is Networking.
These users can use the glue layer interface to avoid misuse of DMAengine
API and when the core gains the needed features they can be converted to
use generic API.
Signed-off-by: Grygorii Strashko <grygorii.strashko@ti.com>
Signed-off-by: Peter Ujfalusi <peter.ujfalusi@ti.com>
---
drivers/dma/ti/Kconfig | 9 +
drivers/dma/ti/Makefile | 1 +
drivers/dma/ti/k3-navss-udma.c | 1133 +++++++++++++++++++++++++++++
drivers/dma/ti/k3-udma-private.c | 123 ++++
drivers/dma/ti/k3-udma.c | 3 +
drivers/dma/ti/k3-udma.h | 30 +
include/linux/dma/k3-navss-udma.h | 152 ++++
7 files changed, 1451 insertions(+)
create mode 100644 drivers/dma/ti/k3-navss-udma.c
create mode 100644 drivers/dma/ti/k3-udma-private.c
create mode 100644 include/linux/dma/k3-navss-udma.h
diff --git a/drivers/dma/ti/Kconfig b/drivers/dma/ti/Kconfig
index db73371de206..23df9e0a1597 100644
--- a/drivers/dma/ti/Kconfig
+++ b/drivers/dma/ti/Kconfig
@@ -45,5 +45,14 @@ config TI_K3_NAVSS_UDMA
Enable support for the TI UDMA (Unified DMA) controller. This
DMA engine is used in AM65x.
+config TI_K3_NAVSS_UDMA_GLUE_LAYER
+ tristate "Texas Instruments UDMA Glue layer for non DMAengine users"
+ depends on ARCH_K3 || COMPILE_TEST
+ depends on TI_K3_NAVSS_UDMA
+ default y
+ help
+ Say y here to support the K3 NAVSS DMA glue interface
+ If unsure, say N.
+
config TI_DMA_CROSSBAR
bool
diff --git a/drivers/dma/ti/Makefile b/drivers/dma/ti/Makefile
index 911f7c723979..c2533360c0de 100644
--- a/drivers/dma/ti/Makefile
+++ b/drivers/dma/ti/Makefile
@@ -3,4 +3,5 @@ obj-$(CONFIG_TI_CPPI41) += cppi41.o
obj-$(CONFIG_TI_EDMA) += edma.o
obj-$(CONFIG_DMA_OMAP) += omap-dma.o
obj-$(CONFIG_TI_K3_NAVSS_UDMA) += k3-udma.o
+obj-$(CONFIG_TI_K3_NAVSS_UDMA_GLUE_LAYER) += k3-navss-udma.o
obj-$(CONFIG_TI_DMA_CROSSBAR) += dma-crossbar.o
diff --git a/drivers/dma/ti/k3-navss-udma.c b/drivers/dma/ti/k3-navss-udma.c
new file mode 100644
index 000000000000..d1613febc86b
--- /dev/null
+++ b/drivers/dma/ti/k3-navss-udma.c
@@ -0,0 +1,1133 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * K3 NAVSS DMA glue interface
+ *
+ * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com
+ *
+ */
+
+#include <linux/atomic.h>
+#include <linux/delay.h>
+#include <linux/dma-mapping.h>
+#include <linux/io.h>
+#include <linux/module.h>
+#include <linux/of.h>
+#include <linux/platform_device.h>
+#include <linux/pm_runtime.h>
+#include <dt-bindings/dma/k3-udma.h>
+#include <linux/irqchip/irq-ti-sci-inta.h>
+#include <linux/soc/ti/cppi5.h>
+#include <linux/soc/ti/k3-navss-ringacc.h>
+#include <linux/dma/k3-navss-udma.h>
+
+#include "k3-udma.h"
+
+struct k3_nav_udmax_common {
+ struct device *dev;
+ struct udma_dev *udmax;
+ const struct udma_tisci_rm *tisci_rm;
+ struct k3_nav_ringacc *ringacc;
+ u32 src_thread;
+ u32 dst_thread;
+
+ u32 hdesc_size;
+ bool epib;
+ u32 psdata_size;
+ u32 swdata_size;
+};
+
+struct k3_nav_udmax_tx_channel {
+ struct k3_nav_udmax_common common;
+
+ struct udma_tchan *udma_tchanx;
+ int udma_tchan_id;
+ bool need_tisci_free;
+
+ struct k3_nav_ring *ringtx;
+ struct k3_nav_ring *ringtxcq;
+
+ bool psil_paired;
+
+ unsigned int virq;
+
+ atomic_t free_pkts;
+ bool tx_pause_on_err;
+ bool tx_filt_einfo;
+ bool tx_filt_pswords;
+ bool tx_supr_tdpkt;
+};
+
+/**
+ * k3_nav_udmax_rx_flow - UDMA RX flow context data
+ *
+ */
+struct k3_nav_udmax_rx_flow {
+ struct udma_rflow *udma_rflow;
+ int udma_rflow_id;
+ struct k3_nav_ring *ringrx;
+ struct k3_nav_ring *ringrxfdq;
+
+ unsigned int virq;
+};
+
+struct k3_nav_udmax_rx_channel {
+ struct k3_nav_udmax_common common;
+
+ struct udma_rchan *udma_rchanx;
+ int udma_rchan_id;
+ bool need_tisci_free;
+
+ bool psil_paired;
+
+ u32 swdata_size;
+ int flow_id_base;
+
+ struct k3_nav_udmax_rx_flow *flows;
+ u32 flow_num;
+ u32 flows_ready;
+};
+
+#define K3_UDMAX_TDOWN_TIMEOUT_US 1000
+
+static int of_k3_nav_udmax_parse(struct device_node *udmax_np,
+ struct k3_nav_udmax_common *common)
+{
+ common->ringacc = of_k3_nav_ringacc_get_by_phandle(
+ udmax_np, "ti,ringacc");
+ if (IS_ERR(common->ringacc))
+ return PTR_ERR(common->ringacc);
+
+ common->udmax = of_xudma_dev_get(udmax_np, NULL);
+ if (IS_ERR(common->udmax))
+ return PTR_ERR(common->udmax);
+
+ common->tisci_rm = xudma_dev_get_tisci_rm(common->udmax);
+
+ return 0;
+}
+
+static int of_k3_nav_udmax_parse_chn(
+ struct device_node *chn_np,
+ const char *name,
+ struct k3_nav_udmax_common *common,
+ bool tx_chn)
+{
+ struct device_node *psil_cfg_node;
+ struct device_node *ch_cfg_node;
+ struct of_phandle_args dma_spec;
+ int index, ret = 0;
+ char prop[50];
+ u32 val;
+
+ if (unlikely(!name))
+ return -EINVAL;
+
+ index = of_property_match_string(chn_np, "dma-names", name);
+ if (index < 0)
+ return index;
+
+ if (of_parse_phandle_with_args(chn_np, "dmas", "#dma-cells", index,
+ &dma_spec))
+ return -ENOENT;
+
+ if (tx_chn && dma_spec.args[2] != UDMA_DIR_TX) {
+ ret = -EINVAL;
+ goto out_put_spec;
+ }
+
+ if (!tx_chn && dma_spec.args[2] != UDMA_DIR_RX) {
+ ret = -EINVAL;
+ goto out_put_spec;
+ }
+
+ /* get psil cfg node */
+ psil_cfg_node = of_find_node_by_phandle(dma_spec.args[0]);
+ if (!psil_cfg_node) {
+ ret = -ENOENT;
+ goto out_put_spec;
+ }
+
+ snprintf(prop, sizeof(prop), "ti,psil-config%u", dma_spec.args[1]);
+ ch_cfg_node = of_find_node_by_name(psil_cfg_node, prop);
+ if (!ch_cfg_node) {
+ dev_err(common->dev,
+ "Channel %u configuration node is missing\n",
+ dma_spec.args[1]);
+ goto out_put_psil_cfg;
+ }
+
+ common->epib = of_property_read_bool(ch_cfg_node, "ti,needs-epib");
+
+ if (!of_property_read_u32(ch_cfg_node, "ti,psd-size", &val))
+ common->psdata_size = val;
+
+ ret = of_property_read_u32(psil_cfg_node, "ti,psil-base", &val);
+ if (ret) {
+ dev_err(common->dev, "ti,psil-base is missing %d\n", ret);
+ goto out_ch_cfg;
+ }
+
+ if (tx_chn)
+ common->dst_thread = val + dma_spec.args[1];
+ else
+ common->src_thread = val + dma_spec.args[1];
+ ret = of_k3_nav_udmax_parse(dma_spec.np, common);
+
+out_ch_cfg:
+ of_node_put(ch_cfg_node);
+out_put_psil_cfg:
+ of_node_put(psil_cfg_node);
+out_put_spec:
+ of_node_put(dma_spec.np);
+ return ret;
+};
+
+static void k3_nav_udmax_dump_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ struct device *dev = tx_chn->common.dev;
+
+ dev_dbg(dev, "dump_tx_chn:\n"
+ "udma_tchan_id: %d\n"
+ "src_thread: %08x\n"
+ "dst_thread: %08x\n",
+ tx_chn->udma_tchan_id,
+ tx_chn->common.src_thread,
+ tx_chn->common.dst_thread);
+}
+
+static void k3_nav_udmax_dump_tx_rt_chn(struct k3_nav_udmax_tx_channel *chn,
+ char *mark)
+{
+ struct device *dev = chn->common.dev;
+
+ dev_dbg(dev, "=== dump ===> %s\n", mark);
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_CTL_REG,
+ xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_PEER_RT_EN_REG,
+ xudma_tchanrt_read(chn->udma_tchanx,
+ UDMA_TCHAN_RT_PEER_RT_EN_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_PCNT_REG,
+ xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_PCNT_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_BCNT_REG,
+ xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_BCNT_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_TCHAN_RT_SBCNT_REG,
+ xudma_tchanrt_read(chn->udma_tchanx, UDMA_TCHAN_RT_SBCNT_REG));
+}
+
+static int k3_nav_udmax_cfg_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ const struct udma_tisci_rm *tisci_rm = tx_chn->common.tisci_rm;
+ struct ti_sci_msg_rm_udmap_tx_ch_cfg req;
+
+ memset(&req, 0, sizeof(req));
+
+ req.valid_params = TI_SCI_MSG_VALUE_RM_UDMAP_CH_PAUSE_ON_ERR_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_EINFO_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_FILT_PSWORDS_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_TX_SUPR_TDPKT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID;
+ req.nav_id = tisci_rm->tisci_dev_id;
+ req.index = tx_chn->udma_tchan_id;
+ if (tx_chn->tx_pause_on_err)
+ req.tx_pause_on_err = 1;
+ if (tx_chn->tx_filt_einfo)
+ req.tx_filt_einfo = 1;
+ if (tx_chn->tx_filt_pswords)
+ req.tx_filt_pswords = 1;
+ req.tx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_PKT_PBRR;
+ if (tx_chn->tx_supr_tdpkt)
+ req.tx_supr_tdpkt = 1;
+ req.tx_fetch_size = tx_chn->common.hdesc_size >> 2;
+ req.txcq_qnum = k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq);
+
+ return tisci_rm->tisci_udmap_ops->tx_ch_cfg(tisci_rm->tisci, &req);
+}
+
+struct k3_nav_udmax_tx_channel *k3_nav_udmax_request_tx_chn(
+ struct device *dev,
+ const char *name,
+ struct k3_nav_udmax_tx_channel_cfg *cfg)
+{
+ struct k3_nav_udmax_tx_channel *tx_chn;
+ int ret;
+
+ tx_chn = devm_kzalloc(dev, sizeof(*tx_chn), GFP_KERNEL);
+ if (!tx_chn)
+ return ERR_PTR(-ENOMEM);
+
+ tx_chn->common.dev = dev;
+ tx_chn->common.swdata_size = cfg->swdata_size;
+ tx_chn->tx_pause_on_err = cfg->tx_pause_on_err;
+ tx_chn->tx_filt_einfo = cfg->tx_filt_einfo;
+ tx_chn->tx_filt_pswords = cfg->tx_filt_pswords;
+ tx_chn->tx_supr_tdpkt = cfg->tx_supr_tdpkt;
+
+ /* parse of udmap channel */
+ ret = of_k3_nav_udmax_parse_chn(dev->of_node, name,
+ &tx_chn->common, true);
+ if (ret)
+ goto err;
+
+ tx_chn->common.hdesc_size = knav_udmap_hdesc_calc_size(
+ tx_chn->common.epib,
+ tx_chn->common.psdata_size,
+ tx_chn->common.swdata_size);
+
+ /* request and cfg UDMAP TX channel */
+ tx_chn->udma_tchanx = xudma_tchan_get(tx_chn->common.udmax, -1);
+ if (IS_ERR(tx_chn->udma_tchanx)) {
+ ret = PTR_ERR(tx_chn->udma_tchanx);
+ dev_err(dev, "UDMAX tchanx get err %d\n", ret);
+ goto err;
+ }
+ tx_chn->udma_tchan_id = xudma_tchan_get_id(tx_chn->udma_tchanx);
+
+ atomic_set(&tx_chn->free_pkts, cfg->txcq_cfg.size);
+
+ /* request and cfg rings */
+ tx_chn->ringtx = k3_nav_ringacc_request_ring(tx_chn->common.ringacc,
+ tx_chn->udma_tchan_id, 0);
+ if (!tx_chn->ringtx) {
+ ret = -ENODEV;
+ dev_err(dev, "Failed to get TX ring %u\n",
+ tx_chn->udma_tchan_id);
+ goto err;
+ }
+
+ tx_chn->ringtxcq = k3_nav_ringacc_request_ring(tx_chn->common.ringacc,
+ -1, 0);
+ if (!tx_chn->ringtxcq) {
+ ret = -ENODEV;
+ dev_err(dev, "Failed to get TXCQ ring\n");
+ goto err;
+ }
+
+ ret = k3_nav_ringacc_ring_cfg(tx_chn->ringtx, &cfg->tx_cfg);
+ if (ret) {
+ dev_err(dev, "Failed to cfg ringtx %d\n", ret);
+ goto err;
+ }
+
+ ret = k3_nav_ringacc_ring_cfg(tx_chn->ringtxcq, &cfg->txcq_cfg);
+ if (ret) {
+ dev_err(dev, "Failed to cfg ringtx %d\n", ret);
+ goto err;
+ }
+
+ /* request and cfg psi-l */
+ tx_chn->common.src_thread =
+ xudma_dev_get_psil_base(tx_chn->common.udmax) +
+ tx_chn->udma_tchan_id;
+
+ tx_chn->need_tisci_free = false;
+ ret = k3_nav_udmax_cfg_tx_chn(tx_chn);
+ if (ret) {
+ dev_err(dev, "Failed to cfg tchan %d\n", ret);
+ goto err;
+ }
+
+ tx_chn->need_tisci_free = true;
+
+ ret = xudma_navss_psil_pair(tx_chn->common.udmax,
+ tx_chn->common.src_thread,
+ tx_chn->common.dst_thread);
+ if (ret) {
+ dev_err(dev, "PSI-L request err %d\n", ret);
+ goto err;
+ }
+
+ tx_chn->psil_paired = true;
+
+ k3_nav_udmax_dump_tx_chn(tx_chn);
+
+ return tx_chn;
+
+err:
+ k3_nav_udmax_release_tx_chn(tx_chn);
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_request_tx_chn);
+
+void k3_nav_udmax_release_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ if (tx_chn->psil_paired) {
+ xudma_navss_psil_unpair(tx_chn->common.udmax,
+ tx_chn->common.src_thread,
+ tx_chn->common.dst_thread);
+ tx_chn->psil_paired = false;
+ }
+
+ if (!IS_ERR_OR_NULL(tx_chn->common.udmax)) {
+ if (tx_chn->need_tisci_free) {
+ tx_chn->need_tisci_free = false;
+ }
+
+ if (!IS_ERR_OR_NULL(tx_chn->udma_tchanx))
+ xudma_tchan_put(tx_chn->common.udmax,
+ tx_chn->udma_tchanx);
+
+ xudma_dev_put(tx_chn->common.udmax);
+ }
+
+ if (tx_chn->ringtxcq)
+ k3_nav_ringacc_ring_free(tx_chn->ringtxcq);
+
+ if (tx_chn->ringtx)
+ k3_nav_ringacc_ring_free(tx_chn->ringtx);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_release_tx_chn);
+
+int k3_nav_udmax_push_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn,
+ struct knav_udmap_host_desc_t *desc_tx,
+ dma_addr_t desc_dma)
+{
+ u32 ringtxcq_id;
+
+ if (!atomic_add_unless(&tx_chn->free_pkts, -1, 0))
+ return -ENOMEM;
+
+ ringtxcq_id = k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq);
+ knav_udmap_desc_set_retpolicy(&desc_tx->hdr, 0,
+ ringtxcq_id);
+
+ return k3_nav_ringacc_ring_push(tx_chn->ringtx, &desc_dma);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_push_tx_chn);
+
+int k3_nav_udmax_pop_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn,
+ dma_addr_t *desc_dma)
+{
+ int ret;
+
+ ret = k3_nav_ringacc_ring_pop(tx_chn->ringtxcq, desc_dma);
+ if (!ret)
+ atomic_inc(&tx_chn->free_pkts);
+
+ return ret;
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_pop_tx_chn);
+
+int k3_nav_udmax_enable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ u32 txrt_ctl;
+
+ txrt_ctl = UDMA_PEER_RT_EN_ENABLE;
+ xudma_tchanrt_write(tx_chn->udma_tchanx,
+ UDMA_TCHAN_RT_PEER_RT_EN_REG,
+ txrt_ctl);
+
+ txrt_ctl = xudma_tchanrt_read(tx_chn->udma_tchanx,
+ UDMA_TCHAN_RT_CTL_REG);
+ txrt_ctl |= UDMA_CHAN_RT_CTL_EN;
+ xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG,
+ txrt_ctl);
+
+ k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn en");
+ return 0;
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_enable_tx_chn);
+
+void k3_nav_udmax_disable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn dis1");
+
+ xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG, 0);
+
+ xudma_tchanrt_write(tx_chn->udma_tchanx,
+ UDMA_TCHAN_RT_PEER_RT_EN_REG, 0);
+ k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn dis2");
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_disable_tx_chn);
+
+void k3_nav_udmax_tdown_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn,
+ bool sync)
+{
+ int i = 0;
+ u32 val;
+
+ k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn tdown1");
+
+ xudma_tchanrt_write(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG,
+ UDMA_CHAN_RT_CTL_EN | UDMA_CHAN_RT_CTL_TDOWN);
+
+ val = xudma_tchanrt_read(tx_chn->udma_tchanx, UDMA_TCHAN_RT_CTL_REG);
+
+ while (sync && (val & UDMA_CHAN_RT_CTL_EN)) {
+ val = xudma_tchanrt_read(tx_chn->udma_tchanx,
+ UDMA_TCHAN_RT_CTL_REG);
+ udelay(1);
+ if (i > K3_UDMAX_TDOWN_TIMEOUT_US) {
+ dev_err(tx_chn->common.dev, "TX tdown timeout\n");
+ break;
+ }
+ i++;
+ }
+
+ val = xudma_tchanrt_read(tx_chn->udma_tchanx,
+ UDMA_TCHAN_RT_PEER_RT_EN_REG);
+ if (sync && (val & UDMA_PEER_RT_EN_ENABLE))
+ dev_err(tx_chn->common.dev, "TX tdown peer not stopped\n");
+ k3_nav_udmax_dump_tx_rt_chn(tx_chn, "txchn tdown2");
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_tdown_tx_chn);
+
+void k3_nav_udmax_reset_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn, void *data,
+ void (*cleanup)(void *data, dma_addr_t desc_dma))
+{
+ dma_addr_t desc_dma;
+ int occ_tx, i, ret;
+
+ /* reset TXCQ as it is not input for udma - expected to be empty */
+ if (tx_chn->ringtxcq)
+ k3_nav_ringacc_ring_reset(tx_chn->ringtxcq);
+
+ /*
+ * TXQ reset need to be special way as it is input for udma and its
+ * state cached by udma, so:
+ * 1) save TXQ occ
+ * 2) clean up TXQ and call callback .cleanup() for each desc
+ * 3) reset TXQ in a special way
+ */
+ occ_tx = k3_nav_ringacc_ring_get_occ(tx_chn->ringtx);
+ dev_dbg(tx_chn->common.dev, "TX reset occ_tx %u\n", occ_tx);
+
+ for (i = 0; i < occ_tx; i++) {
+ ret = k3_nav_ringacc_ring_pop(tx_chn->ringtx, &desc_dma);
+ if (ret) {
+ dev_err(tx_chn->common.dev, "TX reset pop %d\n", ret);
+ break;
+ }
+ cleanup(data, desc_dma);
+ }
+
+ k3_nav_ringacc_ring_reset_dma(tx_chn->ringtx, occ_tx);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_reset_tx_chn);
+
+u32 k3_nav_udmax_tx_get_hdesc_size(struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ return tx_chn->common.hdesc_size;
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_get_hdesc_size);
+
+u32 k3_nav_udmax_tx_get_txcq_id(struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ return k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_get_txcq_id);
+
+int k3_nav_udmax_tx_get_irq(struct k3_nav_udmax_tx_channel *tx_chn,
+ unsigned int *irq, u32 flags, bool share,
+ struct k3_nav_udmax_tx_channel *tx_chn_share)
+{
+ unsigned int virq = 0;
+
+ if (share && tx_chn_share)
+ virq = tx_chn_share->virq;
+
+ tx_chn->virq = ti_sci_inta_register_event(
+ tx_chn->common.dev,
+ k3_nav_ringacc_get_tisci_dev_id(tx_chn->ringtxcq),
+ k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq),
+ virq, flags);
+ if (tx_chn->virq <= 0)
+ return -ENODEV;
+
+ *irq = tx_chn->virq;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_get_irq);
+
+void k3_nav_udmax_tx_put_irq(struct k3_nav_udmax_tx_channel *tx_chn)
+{
+ if (tx_chn->virq <= 0)
+ return;
+
+ ti_sci_inta_unregister_event(
+ tx_chn->common.dev,
+ k3_nav_ringacc_get_tisci_dev_id(tx_chn->ringtxcq),
+ k3_nav_ringacc_get_ring_id(tx_chn->ringtxcq),
+ tx_chn->virq);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_tx_put_irq);
+
+static int k3_nav_udmax_cfg_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn)
+{
+ const struct udma_tisci_rm *tisci_rm = rx_chn->common.tisci_rm;
+ struct ti_sci_msg_rm_udmap_rx_ch_cfg req;
+ int ret;
+
+ memset(&req, 0, sizeof(req));
+
+ req.valid_params = TI_SCI_MSG_VALUE_RM_UDMAP_CH_FETCH_SIZE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CQ_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_CHAN_TYPE_VALID;
+
+ req.nav_id = tisci_rm->tisci_dev_id;
+ req.index = rx_chn->udma_rchan_id;
+ req.rx_fetch_size = rx_chn->common.hdesc_size >> 2;
+ /*
+ * TODO: we can't support rxcq_qnum/RCHAN[a]_RCQ cfg with current sysfw
+ * and udmax impl, so just configure it to invalid value.
+ * req.rxcq_qnum = k3_nav_ringacc_get_ring_id(rx_chn->flows[0].ringrx);
+ */
+ req.rxcq_qnum = 0xFFFF;
+ if (rx_chn->flow_num && rx_chn->flow_id_base != rx_chn->udma_rchan_id) {
+ /* Default flow + extra ones */
+ req.flowid_start = rx_chn->flow_id_base;
+ req.flowid_cnt = rx_chn->flow_num;
+ req.valid_params |=
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_FLOWID_START_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_CH_RX_FLOWID_CNT_VALID;
+ }
+ req.rx_chan_type = TI_SCI_RM_UDMAP_CHAN_TYPE_PKT_PBRR;
+
+ ret = tisci_rm->tisci_udmap_ops->rx_ch_cfg(tisci_rm->tisci, &req);
+ if (ret)
+ dev_err(rx_chn->common.dev, "rchan%d cfg failed %d\n",
+ rx_chn->udma_rchan_id, ret);
+
+ return ret;
+}
+
+static void k3_nav_udmax_release_rx_flow(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num)
+{
+ struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num];
+
+ if (IS_ERR_OR_NULL(flow->udma_rflow))
+ return;
+
+ if (flow->ringrxfdq)
+ k3_nav_ringacc_ring_free(flow->ringrxfdq);
+
+ if (flow->ringrx)
+ k3_nav_ringacc_ring_free(flow->ringrx);
+
+ xudma_rflow_put(rx_chn->common.udmax, flow->udma_rflow);
+ rx_chn->flows_ready--;
+}
+
+static int k3_nav_udmax_cfg_rx_flow(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_idx,
+ struct k3_nav_udmax_rx_flow_cfg *flow_cfg)
+{
+ struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_idx];
+ const struct udma_tisci_rm *tisci_rm = rx_chn->common.tisci_rm;
+ struct device *dev = rx_chn->common.dev;
+ struct ti_sci_msg_rm_udmap_flow_cfg req;
+ int rx_ring_id;
+ int rx_ringfdq_id;
+ int ret = 0;
+
+ flow->udma_rflow = xudma_rflow_get(rx_chn->common.udmax,
+ flow->udma_rflow_id);
+ if (IS_ERR(flow->udma_rflow)) {
+ ret = PTR_ERR(flow->udma_rflow);
+ dev_err(dev, "UDMAX rflow get err %d\n", ret);
+ goto err;
+ }
+
+ if (flow->udma_rflow_id != xudma_rflow_get_id(flow->udma_rflow)) {
+ xudma_rflow_put(rx_chn->common.udmax, flow->udma_rflow);
+ return -ENODEV;
+ }
+
+ /* request and cfg rings */
+ flow->ringrx = k3_nav_ringacc_request_ring(rx_chn->common.ringacc,
+ flow_cfg->ring_rxq_id, 0);
+ if (!flow->ringrx) {
+ ret = -ENODEV;
+ dev_err(dev, "Failed to get RX ring\n");
+ goto err;
+ }
+
+ flow->ringrxfdq = k3_nav_ringacc_request_ring(rx_chn->common.ringacc,
+ flow_cfg->ring_rxfdq0_id,
+ 0);
+ if (!flow->ringrxfdq) {
+ ret = -ENODEV;
+ dev_err(dev, "Failed to get RXFDQ ring\n");
+ goto err;
+ }
+
+ ret = k3_nav_ringacc_ring_cfg(flow->ringrx, &flow_cfg->rx_cfg);
+ if (ret) {
+ dev_err(dev, "Failed to cfg ringrx %d\n", ret);
+ goto err;
+ }
+
+ ret = k3_nav_ringacc_ring_cfg(flow->ringrxfdq, &flow_cfg->rxfdq_cfg);
+ if (ret) {
+ dev_err(dev, "Failed to cfg ringrxfdq %d\n", ret);
+ goto err;
+ }
+
+ rx_ring_id = k3_nav_ringacc_get_ring_id(flow->ringrx);
+ rx_ringfdq_id = k3_nav_ringacc_get_ring_id(flow->ringrxfdq);
+
+ memset(&req, 0, sizeof(req));
+
+ req.valid_params =
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_EINFO_PRESENT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_PSINFO_PRESENT_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_ERROR_HANDLING_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DESC_TYPE_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_HI_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_SRC_TAG_LO_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_HI_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_DEST_TAG_LO_SEL_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ0_SZ0_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ1_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ2_QNUM_VALID |
+ TI_SCI_MSG_VALUE_RM_UDMAP_FLOW_FDQ3_QNUM_VALID;
+ req.nav_id = tisci_rm->tisci_dev_id;
+ req.flow_index = flow->udma_rflow_id;
+ if (rx_chn->common.epib)
+ req.rx_einfo_present = 1;
+ if (rx_chn->common.psdata_size)
+ req.rx_psinfo_present = 1;
+ if (flow_cfg->rx_error_handling)
+ req.rx_error_handling = 1;
+ req.rx_desc_type = 0;
+ req.rx_dest_qnum = rx_ring_id;
+ req.rx_src_tag_hi_sel = 0;
+ req.rx_src_tag_lo_sel = flow_cfg->src_tag_lo_sel;
+ req.rx_dest_tag_hi_sel = 0;
+ req.rx_dest_tag_lo_sel = 0;
+ req.rx_fdq0_sz0_qnum = rx_ringfdq_id;
+ req.rx_fdq1_qnum = rx_ringfdq_id;
+ req.rx_fdq2_qnum = rx_ringfdq_id;
+ req.rx_fdq3_qnum = rx_ringfdq_id;
+
+ ret = tisci_rm->tisci_udmap_ops->rx_flow_cfg(tisci_rm->tisci, &req);
+ if (ret) {
+ dev_err(dev, "flow%d config failed: %d\n", flow->udma_rflow_id,
+ ret);
+ goto err;
+ }
+
+ rx_chn->flows_ready++;
+
+ return 0;
+err:
+ k3_nav_udmax_release_rx_flow(rx_chn, flow_idx);
+ return ret;
+}
+
+static void k3_nav_udmax_dump_rx_chn(struct k3_nav_udmax_rx_channel *chn)
+{
+ struct device *dev = chn->common.dev;
+
+ dev_dbg(dev, "dump_rx_chn:\n"
+ "udma_rchan_id: %d\n"
+ "src_thread: %08x\n"
+ "dst_thread: %08x\n"
+ "epib: %d\n"
+ "hdesc_size: %u\n"
+ "psdata_size: %u\n"
+ "swdata_size: %u\n"
+ "flow_id_base: %d\n"
+ "flow_num: %d\n",
+ chn->udma_rchan_id,
+ chn->common.src_thread,
+ chn->common.dst_thread,
+ chn->common.epib,
+ chn->common.hdesc_size,
+ chn->common.psdata_size,
+ chn->common.swdata_size,
+ chn->flow_id_base,
+ chn->flow_num);
+}
+
+static void k3_nav_udmax_dump_rx_rt_chn(struct k3_nav_udmax_rx_channel *chn,
+ char *mark)
+{
+ struct device *dev = chn->common.dev;
+
+ dev_dbg(dev, "=== dump ===> %s\n", mark);
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_CTL_REG,
+ xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ xudma_rchanrt_read(chn->udma_rchanx,
+ UDMA_RCHAN_RT_PEER_RT_EN_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_PCNT_REG,
+ xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_PCNT_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_BCNT_REG,
+ xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_BCNT_REG));
+ dev_dbg(dev, "0x%08X: %08X\n", UDMA_RCHAN_RT_SBCNT_REG,
+ xudma_rchanrt_read(chn->udma_rchanx, UDMA_RCHAN_RT_SBCNT_REG));
+}
+
+struct k3_nav_udmax_rx_channel *k3_nav_udmax_request_rx_chn(
+ struct device *dev,
+ const char *name,
+ struct k3_nav_udmax_rx_channel_cfg *cfg)
+{
+ struct k3_nav_udmax_rx_channel *rx_chn;
+ int ret, i;
+
+ if (cfg->flow_id_num <= 0)
+ return ERR_PTR(-EINVAL);
+
+ if (cfg->flow_id_num != 1 &&
+ (cfg->def_flow_cfg || cfg->flow_id_use_rxchan_id))
+ return ERR_PTR(-EINVAL);
+
+ rx_chn = devm_kzalloc(dev, sizeof(*rx_chn), GFP_KERNEL);
+ if (!rx_chn)
+ return ERR_PTR(-ENOMEM);
+
+ rx_chn->common.dev = dev;
+ rx_chn->common.swdata_size = cfg->swdata_size;
+
+ /* parse of udmap channel */
+ ret = of_k3_nav_udmax_parse_chn(dev->of_node, name,
+ &rx_chn->common, false);
+ if (ret)
+ goto err;
+
+ rx_chn->common.hdesc_size = knav_udmap_hdesc_calc_size(
+ rx_chn->common.epib,
+ rx_chn->common.psdata_size,
+ rx_chn->common.swdata_size);
+
+ /* request and cfg UDMAP RX channel */
+ rx_chn->udma_rchanx = xudma_rchan_get(rx_chn->common.udmax, -1);
+ if (IS_ERR(rx_chn->udma_rchanx)) {
+ ret = PTR_ERR(rx_chn->udma_rchanx);
+ dev_err(dev, "UDMAX rchanx get err %d\n", ret);
+ goto err;
+ }
+ rx_chn->udma_rchan_id = xudma_rchan_get_id(rx_chn->udma_rchanx);
+
+ rx_chn->flow_num = cfg->flow_id_num;
+ rx_chn->flow_id_base = cfg->flow_id_base;
+
+ /* Use RX channel id as flow id: target dev can't generate flow_id */
+ if (cfg->flow_id_use_rxchan_id)
+ rx_chn->flow_id_base = rx_chn->udma_rchan_id;
+
+ rx_chn->flows = devm_kcalloc(dev, rx_chn->flow_num,
+ sizeof(*rx_chn->flows), GFP_KERNEL);
+ if (!rx_chn->flows) {
+ ret = -ENOMEM;
+ goto err;
+ }
+
+ /* Reserve range of RX flows */
+ if (!cfg->flow_id_use_rxchan_id) {
+ ret = xudma_reserve_rflow_range(rx_chn->common.udmax,
+ rx_chn->flow_id_base,
+ rx_chn->flow_num);
+ if (ret < 0) {
+ dev_err(dev, "UDMAX reserve_rflow get err %d\n", ret);
+ goto err;
+ }
+ rx_chn->flow_id_base = ret;
+ }
+
+ for (i = 0; i < rx_chn->flow_num; i++)
+ rx_chn->flows[i].udma_rflow_id = rx_chn->flow_id_base + i;
+
+ /* request and cfg psi-l */
+ rx_chn->common.dst_thread =
+ xudma_dev_get_psil_base(rx_chn->common.udmax) +
+ rx_chn->udma_rchan_id;
+
+ rx_chn->need_tisci_free = false;
+ ret = k3_nav_udmax_cfg_rx_chn(rx_chn);
+ if (ret) {
+ dev_err(dev, "Failed to cfg rchan %d\n", ret);
+ goto err;
+ }
+ rx_chn->need_tisci_free = true;
+
+ /* init default RX flow only if flow_num = 1 */
+ if (cfg->def_flow_cfg) {
+ ret = k3_nav_udmax_cfg_rx_flow(rx_chn, 0, cfg->def_flow_cfg);
+ if (ret)
+ goto err;
+ }
+
+ if (!cfg->skip_psil) {
+ ret = xudma_navss_psil_pair(rx_chn->common.udmax,
+ rx_chn->common.src_thread,
+ rx_chn->common.dst_thread);
+ if (ret) {
+ dev_err(dev, "PSI-L request err %d\n", ret);
+ goto err;
+ }
+
+ rx_chn->psil_paired = true;
+ }
+
+ k3_nav_udmax_dump_rx_chn(rx_chn);
+
+ return rx_chn;
+
+err:
+ k3_nav_udmax_release_rx_chn(rx_chn);
+ return ERR_PTR(ret);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_request_rx_chn);
+
+void k3_nav_udmax_release_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn)
+{
+ int i;
+
+ if (rx_chn->psil_paired) {
+ xudma_navss_psil_unpair(rx_chn->common.udmax,
+ rx_chn->common.src_thread,
+ rx_chn->common.dst_thread);
+ rx_chn->psil_paired = false;
+ }
+
+ if (IS_ERR_OR_NULL(rx_chn->common.udmax))
+ return;
+
+ for (i = 0; i < rx_chn->flow_num; i++)
+ k3_nav_udmax_release_rx_flow(rx_chn, i);
+
+ if (rx_chn->need_tisci_free) {
+ rx_chn->need_tisci_free = false;
+ }
+
+ xudma_free_rflow_range(rx_chn->common.udmax,
+ rx_chn->flow_id_base, rx_chn->flow_num);
+
+ if (!IS_ERR_OR_NULL(rx_chn->udma_rchanx))
+ xudma_rchan_put(rx_chn->common.udmax,
+ rx_chn->udma_rchanx);
+
+ xudma_dev_put(rx_chn->common.udmax);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_release_rx_chn);
+
+int k3_nav_udmax_rx_flow_init(
+ struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx,
+ struct k3_nav_udmax_rx_flow_cfg *flow_cfg)
+{
+ if (flow_idx >= rx_chn->flow_num)
+ return -EINVAL;
+
+ return k3_nav_udmax_cfg_rx_flow(rx_chn, flow_idx, flow_cfg);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_flow_init);
+
+u32 k3_nav_udmax_rx_flow_get_fdq_id(
+ struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx)
+{
+ struct k3_nav_udmax_rx_flow *flow;
+
+ if (flow_idx >= rx_chn->flow_num)
+ return -EINVAL;
+
+ flow = &rx_chn->flows[flow_idx];
+
+ return k3_nav_ringacc_get_ring_id(flow->ringrxfdq);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_flow_get_fdq_id);
+
+u32 k3_nav_udmax_rx_get_flow_id_base(
+ struct k3_nav_udmax_rx_channel *rx_chn)
+{
+ return rx_chn->flow_id_base;
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_get_flow_id_base);
+
+int k3_nav_udmax_enable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn)
+{
+ u32 rxrt_ctl;
+
+ if (rx_chn->flows_ready < rx_chn->flow_num)
+ return -EINVAL;
+
+ rxrt_ctl = xudma_rchanrt_read(rx_chn->udma_rchanx,
+ UDMA_RCHAN_RT_CTL_REG);
+ rxrt_ctl |= UDMA_CHAN_RT_CTL_EN;
+ xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG,
+ rxrt_ctl);
+
+ xudma_rchanrt_write(rx_chn->udma_rchanx,
+ UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_ENABLE);
+
+ k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt en");
+ return 0;
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_enable_rx_chn);
+
+void k3_nav_udmax_disable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn)
+{
+ k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt dis1");
+
+ xudma_rchanrt_write(rx_chn->udma_rchanx,
+ UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ 0);
+ xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG, 0);
+
+ k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt dis2");
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_disable_rx_chn);
+
+void k3_nav_udmax_tdown_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn,
+ bool sync)
+{
+ int i = 0;
+ u32 val;
+
+ k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt tdown1");
+
+ xudma_rchanrt_write(rx_chn->udma_rchanx, UDMA_RCHAN_RT_PEER_RT_EN_REG,
+ UDMA_PEER_RT_EN_ENABLE | UDMA_PEER_RT_EN_TEARDOWN);
+
+ val = xudma_rchanrt_read(rx_chn->udma_rchanx, UDMA_RCHAN_RT_CTL_REG);
+
+ while (sync && (val & UDMA_CHAN_RT_CTL_EN)) {
+ val = xudma_rchanrt_read(rx_chn->udma_rchanx,
+ UDMA_RCHAN_RT_CTL_REG);
+ udelay(1);
+ if (i > K3_UDMAX_TDOWN_TIMEOUT_US) {
+ dev_err(rx_chn->common.dev, "RX tdown timeout\n");
+ break;
+ }
+ i++;
+ }
+
+ val = xudma_rchanrt_read(rx_chn->udma_rchanx,
+ UDMA_RCHAN_RT_PEER_RT_EN_REG);
+ if (sync && (val & UDMA_PEER_RT_EN_ENABLE))
+ dev_err(rx_chn->common.dev, "TX tdown peer not stopped\n");
+ k3_nav_udmax_dump_rx_rt_chn(rx_chn, "rxrt tdown2");
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_tdown_rx_chn);
+
+void k3_nav_udmax_reset_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num, void *data,
+ void (*cleanup)(void *data, dma_addr_t desc_dma),
+ bool skip_fdq)
+{
+ struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num];
+ struct device *dev = rx_chn->common.dev;
+ dma_addr_t desc_dma;
+ int occ_rx, i, ret;
+
+ /* reset RXCQ as it is not input for udma - expected to be empty */
+ if (flow->ringrx)
+ k3_nav_ringacc_ring_reset(flow->ringrx);
+
+ /* Skip RX FDQ in case one FDQ is used for the set of flows */
+ if (skip_fdq)
+ return;
+
+ /*
+ * RX FDQ reset need to be special way as it is input for udma and its
+ * state cached by udma, so:
+ * 1) save RX FDQ occ
+ * 2) clean up RX FDQ and call callback .cleanup() for each desc
+ * 3) reset RX FDQ in a special way
+ */
+ occ_rx = k3_nav_ringacc_ring_get_occ(flow->ringrxfdq);
+ dev_dbg(dev, "RX reset flow %u occ_tx %u\n", flow_num, occ_rx);
+
+ for (i = 0; i < occ_rx; i++) {
+ ret = k3_nav_ringacc_ring_pop(flow->ringrxfdq, &desc_dma);
+ if (ret) {
+ dev_err(dev, "RX reset pop %d\n", ret);
+ break;
+ }
+ cleanup(data, desc_dma);
+ }
+
+ k3_nav_ringacc_ring_reset_dma(flow->ringrxfdq, occ_rx);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_reset_rx_chn);
+
+int k3_nav_udmax_push_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num,
+ struct knav_udmap_host_desc_t *desc_rx,
+ dma_addr_t desc_dma)
+{
+ struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num];
+
+ return k3_nav_ringacc_ring_push(flow->ringrxfdq, &desc_dma);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_push_rx_chn);
+
+int k3_nav_udmax_pop_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num,
+ dma_addr_t *desc_dma)
+{
+ struct k3_nav_udmax_rx_flow *flow = &rx_chn->flows[flow_num];
+
+ return k3_nav_ringacc_ring_pop(flow->ringrx, desc_dma);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_pop_rx_chn);
+
+int k3_nav_udmax_rx_get_irq(struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num,
+ unsigned int *irq, u32 flags, bool share,
+ u32 flow_num_share)
+{
+ struct k3_nav_udmax_rx_flow *flow, *flow_share;
+ unsigned int virq = 0;
+
+ if (flow_num >= rx_chn->flow_num ||
+ (flow_num_share != -1 && flow_num_share >= rx_chn->flow_num))
+ return -EINVAL;
+
+ flow = &rx_chn->flows[flow_num];
+
+ if (share && flow_num_share != -1) {
+ flow_share = &rx_chn->flows[flow_num_share];
+ virq = flow_share->virq;
+ }
+
+ flow->virq = ti_sci_inta_register_event(
+ rx_chn->common.dev,
+ k3_nav_ringacc_get_tisci_dev_id(flow->ringrx),
+ k3_nav_ringacc_get_ring_id(flow->ringrx),
+ virq, flags);
+ if (flow->virq <= 0)
+ return -ENODEV;
+
+ *irq = flow->virq;
+
+ return 0;
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_get_irq);
+
+void k3_nav_udmax_rx_put_irq(struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num)
+{
+ struct k3_nav_udmax_rx_flow *flow;
+
+ if (flow_num >= rx_chn->flow_num)
+ return;
+
+ flow = &rx_chn->flows[flow_num];
+
+ if (flow->virq <= 0)
+ return;
+
+ ti_sci_inta_unregister_event(
+ rx_chn->common.dev,
+ k3_nav_ringacc_get_tisci_dev_id(flow->ringrx),
+ k3_nav_ringacc_get_ring_id(flow->ringrx), flow->virq);
+}
+EXPORT_SYMBOL_GPL(k3_nav_udmax_rx_put_irq);
diff --git a/drivers/dma/ti/k3-udma-private.c b/drivers/dma/ti/k3-udma-private.c
new file mode 100644
index 000000000000..21bae700f7a2
--- /dev/null
+++ b/drivers/dma/ti/k3-udma-private.c
@@ -0,0 +1,123 @@
+// SPDX-License-Identifier: GPL-2.0
+/*
+ * Copyright (C) 2018 Texas Instruments Incorporated - http://www.ti.com
+ * Author: Peter Ujfalusi <peter.ujfalusi@ti.com>
+ */
+
+int xudma_navss_psil_pair(struct udma_dev *ud, u32 src_thread,u32 dst_thread)
+{
+ return navss_psil_pair(ud, src_thread, dst_thread);
+}
+EXPORT_SYMBOL(xudma_navss_psil_pair);
+
+int xudma_navss_psil_unpair(struct udma_dev *ud, u32 src_thread, u32 dst_thread)
+{
+ return navss_psil_unpair(ud, src_thread, dst_thread);
+}
+EXPORT_SYMBOL(xudma_navss_psil_unpair);
+
+struct udma_dev *of_xudma_dev_get(struct device_node *np, const char *property)
+{
+ struct device_node *udma_node = np;
+ struct platform_device *pdev;
+ struct udma_dev *ud;
+
+ if (property) {
+ udma_node = of_parse_phandle(np, property, 0);
+ if (!udma_node) {
+ pr_err("UDMA node is not found\n");
+ return ERR_PTR(-ENODEV);
+ }
+ }
+
+ pdev = of_find_device_by_node(udma_node);
+ if (!pdev) {
+ pr_err("UDMA device not found\n");
+ return ERR_PTR(-EPROBE_DEFER);
+ }
+
+ of_node_put(udma_node);
+
+ ud = platform_get_drvdata(pdev);
+ if (!ud) {
+ pr_err("UDMA has not been proped\n");
+ return ERR_PTR(-EPROBE_DEFER);
+ }
+
+ pm_runtime_get_sync(&pdev->dev);
+
+ return ud;
+}
+EXPORT_SYMBOL(of_xudma_dev_get);
+
+void xudma_dev_put(struct udma_dev *ud)
+{
+ pm_runtime_put_sync(ud->ddev.dev);
+}
+EXPORT_SYMBOL(xudma_dev_put);
+
+u32 xudma_dev_get_psil_base(struct udma_dev *ud)
+{
+ return ud->psil_base;
+}
+EXPORT_SYMBOL(xudma_dev_get_psil_base);
+
+struct udma_tisci_rm *xudma_dev_get_tisci_rm(struct udma_dev *ud)
+{
+ return &ud->tisci_rm;
+}
+EXPORT_SYMBOL(xudma_dev_get_tisci_rm);
+
+int xudma_reserve_rflow_range(struct udma_dev *ud, int from, int cnt)
+{
+ return __udma_reserve_rflow_range(ud, from, cnt);
+}
+EXPORT_SYMBOL(xudma_reserve_rflow_range);
+
+int xudma_free_rflow_range(struct udma_dev *ud, int from, int cnt)
+{
+ return __udma_free_rflow_range(ud, from, cnt);
+}
+EXPORT_SYMBOL(xudma_free_rflow_range);
+
+#define XUDMA_GET_PUT_RESOURCE(res) \
+struct udma_##res *xudma_##res##_get(struct udma_dev *ud, int id) \
+{ \
+ return __udma_reserve_##res(ud, false, id); \
+} \
+EXPORT_SYMBOL(xudma_##res##_get); \
+ \
+void xudma_##res##_put(struct udma_dev *ud, struct udma_##res *p) \
+{ \
+ clear_bit(p->id, ud->res##_map); \
+} \
+EXPORT_SYMBOL(xudma_##res##_put)
+XUDMA_GET_PUT_RESOURCE(tchan);
+XUDMA_GET_PUT_RESOURCE(rchan);
+XUDMA_GET_PUT_RESOURCE(rflow);
+
+#define XUDMA_GET_RESOURCE_ID(res) \
+int xudma_##res##_get_id(struct udma_##res *p) \
+{ \
+ return p->id; \
+} \
+EXPORT_SYMBOL(xudma_##res##_get_id)
+XUDMA_GET_RESOURCE_ID(tchan);
+XUDMA_GET_RESOURCE_ID(rchan);
+XUDMA_GET_RESOURCE_ID(rflow);
+
+/* Exported register access functions */
+#define XUDMA_RT_IO_FUNCTIONS(res) \
+u32 xudma_##res##rt_read(struct udma_##res *p, int reg) \
+{ \
+ return udma_##res##rt_read(p, reg); \
+} \
+EXPORT_SYMBOL(xudma_##res##rt_read); \
+ \
+void xudma_##res##rt_write(struct udma_##res *p, int reg, u32 val) \
+{ \
+ udma_##res##rt_write(p, reg, val); \
+} \
+EXPORT_SYMBOL(xudma_##res##rt_write)
+XUDMA_RT_IO_FUNCTIONS(tchan);
+XUDMA_RT_IO_FUNCTIONS(rchan);
diff --git a/drivers/dma/ti/k3-udma.c b/drivers/dma/ti/k3-udma.c
index 2a2a2c8b1fc2..49d4c49b8139 100644
--- a/drivers/dma/ti/k3-udma.c
+++ b/drivers/dma/ti/k3-udma.c
@@ -3617,6 +3617,9 @@ static struct platform_driver udma_driver = {
module_platform_driver(udma_driver);
+/* Private interfaces to UDMA */
+#include "k3-udma-private.c"
+
MODULE_ALIAS("platform:ti-udma");
MODULE_DESCRIPTION("TI K3 DMA driver for CPPI 5.0 compliant devices");
MODULE_AUTHOR("Peter Ujfalusi <peter.ujfalusi@ti.com>");
diff --git a/drivers/dma/ti/k3-udma.h b/drivers/dma/ti/k3-udma.h
index 787b9feefe5d..2720d73ff3d3 100644
--- a/drivers/dma/ti/k3-udma.h
+++ b/drivers/dma/ti/k3-udma.h
@@ -129,4 +129,34 @@ struct udma_tisci_rm {
struct ti_sci_resource *rm_ranges[RM_RANGE_LAST];
};
+/* Direct access to UDMA resources */
+int xudma_navss_psil_pair(struct udma_dev *ud, u32 src_thread,u32 dst_thread);
+int xudma_navss_psil_unpair(struct udma_dev *ud, u32 src_thread,
+ u32 dst_thread);
+
+struct udma_dev *of_xudma_dev_get(struct device_node *np, const char *property);
+void xudma_dev_put(struct udma_dev *ud);
+u32 xudma_dev_get_psil_base(struct udma_dev *ud);
+struct udma_tisci_rm *xudma_dev_get_tisci_rm(struct udma_dev *ud);
+
+int xudma_reserve_rflow_range(struct udma_dev *ud, int from, int cnt);
+int xudma_free_rflow_range(struct udma_dev *ud, int from, int cnt);
+
+struct udma_tchan *xudma_tchan_get(struct udma_dev *ud, int id);
+struct udma_rchan *xudma_rchan_get(struct udma_dev *ud, int id);
+struct udma_rflow *xudma_rflow_get(struct udma_dev *ud, int id);
+
+void xudma_tchan_put(struct udma_dev *ud, struct udma_tchan *p);
+void xudma_rchan_put(struct udma_dev *ud, struct udma_rchan *p);
+void xudma_rflow_put(struct udma_dev *ud, struct udma_rflow *p);
+
+int xudma_tchan_get_id(struct udma_tchan *p);
+int xudma_rchan_get_id(struct udma_rchan *p);
+int xudma_rflow_get_id(struct udma_rflow *p);
+
+u32 xudma_tchanrt_read(struct udma_tchan *tchan, int reg);
+void xudma_tchanrt_write(struct udma_tchan *tchan, int reg, u32 val);
+u32 xudma_rchanrt_read(struct udma_rchan *rchan, int reg);
+void xudma_rchanrt_write(struct udma_rchan *rchan, int reg, u32 val);
+
#endif /* K3_UDMA_H_ */
diff --git a/include/linux/dma/k3-navss-udma.h b/include/linux/dma/k3-navss-udma.h
new file mode 100644
index 000000000000..64a8bcbb8703
--- /dev/null
+++ b/include/linux/dma/k3-navss-udma.h
@@ -0,0 +1,152 @@
+/*
+ * K3 NAVSS DMA glue interface
+ *
+ * Copyright (C) 2017 Texas Instruments Incorporated - http://www.ti.com
+ *
+ * This program is free software; you can redistribute it and/or
+ * modify it under the terms of the GNU General Public License
+ * version 2 as published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful, but
+ * WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
+ * General Public License for more details.
+ */
+
+#ifndef K3_NAVSS_UDMA_H_
+#define K3_NAVSS_UDMA_H_
+
+#include <linux/soc/ti/k3-navss-ringacc.h>
+
+struct k3_nav_udmax_tx_channel_cfg {
+ struct k3_nav_ring_cfg tx_cfg;
+ struct k3_nav_ring_cfg txcq_cfg;
+
+ bool tx_pause_on_err;
+ bool tx_filt_einfo;
+ bool tx_filt_pswords;
+ bool tx_supr_tdpkt;
+ u32 swdata_size;
+};
+
+struct k3_nav_udmax_tx_channel;
+
+struct k3_nav_udmax_tx_channel *k3_nav_udmax_request_tx_chn(
+ struct device *dev,
+ const char *name,
+ struct k3_nav_udmax_tx_channel_cfg *cfg);
+
+void k3_nav_udmax_release_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn);
+int k3_nav_udmax_push_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn,
+ struct knav_udmap_host_desc_t *desc_tx,
+ dma_addr_t desc_dma);
+int k3_nav_udmax_pop_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn,
+ dma_addr_t *desc_dma);
+int k3_nav_udmax_enable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn);
+void k3_nav_udmax_disable_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn);
+void k3_nav_udmax_tdown_tx_chn(struct k3_nav_udmax_tx_channel *tx_chn,
+ bool sync);
+void k3_nav_udmax_reset_tx_chn(
+ struct k3_nav_udmax_tx_channel *tx_chn, void *data,
+ void (*cleanup)(void *data, dma_addr_t desc_dma));
+u32 k3_nav_udmax_tx_get_hdesc_size(struct k3_nav_udmax_tx_channel *tx_chn);
+u32 k3_nav_udmax_tx_get_txcq_id(struct k3_nav_udmax_tx_channel *tx_chn);
+int k3_nav_udmax_tx_get_irq(struct k3_nav_udmax_tx_channel *tx_chn,
+ unsigned int *irq, u32 flags, bool share,
+ struct k3_nav_udmax_tx_channel *tx_chn_share);
+void k3_nav_udmax_tx_put_irq(struct k3_nav_udmax_tx_channel *tx_chn);
+
+enum {
+ K3_NAV_UDMAX_SRC_TAG_LO_KEEP = 0,
+ K3_NAV_UDMAX_SRC_TAG_LO_USE_FLOW_REG = 1,
+ K3_NAV_UDMAX_SRC_TAG_LO_USE_REMOTE_FLOW_ID = 2,
+ K3_NAV_UDMAX_SRC_TAG_LO_USE_REMOTE_SRC_TAG = 4,
+};
+
+/**
+ * k3_nav_udmax_rx_flow_cfg - UDMA RX flow cfg
+ *
+ * @rx_cfg: RX ring configuration
+ * @rxfdq_cfg: RX free Host PD ring configuration
+ * @ring_rxq_id: RX ring id (or -1 for any)
+ * @ring_rxfdq0_id: RX free Host PD ring (FDQ) if (or -1 for any)
+ * @rx_error_handling: Rx Error Handling Mode (0 - drop, 1 - re-try)
+ * @src_tag_lo_sel: Rx Source Tag Low Byte Selector in Host PD
+ */
+struct k3_nav_udmax_rx_flow_cfg {
+ struct k3_nav_ring_cfg rx_cfg;
+ struct k3_nav_ring_cfg rxfdq_cfg;
+ int ring_rxq_id;
+ int ring_rxfdq0_id;
+ bool rx_error_handling;
+ int src_tag_lo_sel;
+};
+
+/**
+ * k3_nav_udmax_rx_channel_cfg - UDMA RX channel cfg
+ *
+ * @psdata_size: SW Data is present in Host PD of @swdata_size bytes
+ * @flow_id_base: first flow_id used by channel.
+ * if @flow_id_base = -1 - flow ids range will be allocated
+ * dynamically.
+ * @flow_id_num: number of RX flows used by channel
+ * @flow_id_use_rxchan_id: use RX channel id as flow id,
+ * used only if @flow_id_num = 1
+ * @def_flow_cfg default RX flow configuration,
+ * used only if @flow_id_num = 1
+ */
+struct k3_nav_udmax_rx_channel_cfg {
+ u32 swdata_size;
+ int flow_id_base;
+ int flow_id_num;
+ bool flow_id_use_rxchan_id;
+ bool skip_psil;
+
+ struct k3_nav_udmax_rx_flow_cfg *def_flow_cfg;
+};
+
+struct k3_nav_udmax_rx_channel;
+
+struct k3_nav_udmax_rx_channel *k3_nav_udmax_request_rx_chn(
+ struct device *dev,
+ const char *name,
+ struct k3_nav_udmax_rx_channel_cfg *cfg);
+
+void k3_nav_udmax_release_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn);
+int k3_nav_udmax_enable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn);
+void k3_nav_udmax_disable_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn);
+void k3_nav_udmax_tdown_rx_chn(struct k3_nav_udmax_rx_channel *rx_chn,
+ bool sync);
+int k3_nav_udmax_push_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num,
+ struct knav_udmap_host_desc_t *desc_tx,
+ dma_addr_t desc_dma);
+int k3_nav_udmax_pop_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num,
+ dma_addr_t *desc_dma);
+int k3_nav_udmax_rx_flow_init(
+ struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx,
+ struct k3_nav_udmax_rx_flow_cfg *flow_cfg);
+u32 k3_nav_udmax_rx_flow_get_fdq_id(
+ struct k3_nav_udmax_rx_channel *rx_chn, u32 flow_idx);
+u32 k3_nav_udmax_rx_get_flow_id_base(
+ struct k3_nav_udmax_rx_channel *rx_chn);
+int k3_nav_udmax_rx_get_irq(struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num,
+ unsigned int *irq, u32 flags, bool share,
+ u32 flow_num_share);
+void k3_nav_udmax_rx_put_irq(struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num);
+void k3_nav_udmax_reset_rx_chn(
+ struct k3_nav_udmax_rx_channel *rx_chn,
+ u32 flow_num, void *data,
+ void (*cleanup)(void *data, dma_addr_t desc_dma),
+ bool skip_fdq);
+
+#endif /* K3_NAVSS_UDMA_H_ */
^ permalink raw reply related
* [v1,1/7] dt-bindings: stm32-dma: Add DMA/MDMA chaining support bindings
From: Rob Herring @ 2018-09-26 20:21 UTC (permalink / raw)
To: Pierre-Yves MORDRET
Cc: Vinod Koul, Mark Rutland, Alexandre Torgue, Maxime Coquelin,
Dan Williams, devicetree, dmaengine, linux-arm-kernel,
linux-kernel
On Tue, Sep 11, 2018 at 09:26:54AM +0200, Pierre-Yves MORDRET wrote:
> This patch adds dma bindings to support DMA/MDMA chaining transfer.
> 1 bit is to manage both DMA FIFO Threshold
> 1 bit is to manage DMA/MDMA Chaining features.
> 2 bits are used to specify SDRAM size to use for DMA/MDMA chaining.
> The size in bytes of a certain order is given by the formula:
> (2 ^ order) * PAGE_SIZE.
> The order is given by those 2 bits.
> For cyclic, whether chaining is chosen, any value above 1 can be set :
> SRAM buffer size will rely on period size and not on this DT value.
>
> Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
> ---
> Version history:
> v1:
> * Initial
> ---
> ---
> .../devicetree/bindings/dma/stm32-dma.txt | 32 +++++++++++++++++++---
> 1 file changed, 28 insertions(+), 4 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/dma/stm32-dma.txt b/Documentation/devicetree/bindings/dma/stm32-dma.txt
> index c5f5190..163be09 100644
> --- a/Documentation/devicetree/bindings/dma/stm32-dma.txt
> +++ b/Documentation/devicetree/bindings/dma/stm32-dma.txt
> @@ -17,6 +17,12 @@ Optional properties:
> - resets: Reference to a reset controller asserting the DMA controller
> - st,mem2mem: boolean; if defined, it indicates that the controller supports
> memory-to-memory transfer
> +- dmas: A list of eight dma specifiers, one for each entry in dma-names.
> + Refer to stm32-mdma.txt for more details.
> +- dma-names: should contain "ch0", "ch1", "ch2", "ch3", "ch4", "ch5", "ch6" and
> + "ch7" and represents each STM32 DMA channel connected to a STM32 MDMA one.
> +- memory-region : phandle to a node describing memory to be used for
> + M2M intermediate transfer between DMA and MDMA.
>
> Example:
>
> @@ -36,6 +42,16 @@ Example:
> st,mem2mem;
> resets = <&rcc 150>;
> dma-requests = <8>;
> + dmas = <&mdma1 8 0x10 0x1200000a 0x40026408 0x00000020 1>,
> + <&mdma1 9 0x10 0x1200000a 0x40026408 0x00000800 1>,
> + <&mdma1 10 0x10 0x1200000a 0x40026408 0x00200000 1>,
> + <&mdma1 11 0x10 0x1200000a 0x40026408 0x08000000 1>,
> + <&mdma1 12 0x10 0x1200000a 0x4002640C 0x00000020 1>,
> + <&mdma1 13 0x10 0x1200000a 0x4002640C 0x00000800 1>,
> + <&mdma1 14 0x10 0x1200000a 0x4002640C 0x00200000 1>,
> + <&mdma1 15 0x10 0x1200000a 0x4002640C 0x08000000 1>;
> + dma-names = "ch0", "ch1", "ch2", "ch3", "ch4", "ch5", "ch6", "ch7";
> + memory-region = <&sram_dmapool>;
> };
>
> * DMA client
> @@ -62,13 +78,21 @@ channel: a phandle to the DMA controller plus the following four integer cells:
> 0x1: medium
> 0x2: high
> 0x3: very high
> -4. A 32bit bitfield value specifying DMA features which are device dependent:
> +4. A bitfield value specifying DMA features which are device dependent:
It's still a 32-bit value, right?
> -bit 0-1: DMA FIFO threshold selection
> 0x0: 1/4 full FIFO
> 0x1: 1/2 full FIFO
> 0x2: 3/4 full FIFO
> 0x3: full FIFO
> -
> + -bit 2: Intermediate M2M transfer from/to DDR to/from SRAM throughout MDMA
> + 0: MDMA not used to generate an intermediate M2M transfer
> + 1: MDMA used to generate an intermediate M2M transfer.
> + -bit 3-4: indicated SRAM Buffer size in (2^order)*PAGE_SIZE.
What's PAGE_SIZE? That's not defined for DT.
> + Order is given by those 2 bits starting at 0.
> + Valid only whether Intermediate M2M transfer is set.
> + For cyclic, whether Intermediate M2M transfer is chosen, any value can
> + be set: SRAM buffer size will rely on period size and not on this DT
> + value.
>
> Example:
>
> @@ -77,7 +101,7 @@ Example:
> reg = <0x40011000 0x400>;
> interrupts = <37>;
> clocks = <&clk_pclk2>;
> - dmas = <&dma2 2 4 0x10400 0x3>,
> - <&dma2 7 5 0x10200 0x3>;
> + dmas = <&dma2 2 4 0x10400 0x1>,
> + <&dma2 7 5 0x10200 0x1>;
> dma-names = "rx", "tx";
> };
> --
> 2.7.4
>
^ permalink raw reply
* [v1,2/7] dt-bindings: stm32-dmamux: Add one cell to support DMA/MDMA chain
From: Rob Herring @ 2018-09-26 20:22 UTC (permalink / raw)
To: Pierre-Yves MORDRET
Cc: Vinod Koul, Mark Rutland, Alexandre Torgue, Maxime Coquelin,
Dan Williams, devicetree, dmaengine, linux-arm-kernel,
linux-kernel
On Tue, Sep 11, 2018 at 09:26:55AM +0200, Pierre-Yves MORDRET wrote:
> Add one cell to support DMA/MDMA chaining.
The diff doesn't match the description.
>
> Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
> ---
> Version history:
> v1:
> * Initial
> ---
> ---
> Documentation/devicetree/bindings/dma/stm32-dmamux.txt | 5 +----
> 1 file changed, 1 insertion(+), 4 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
> index 1b893b2..8e092d2 100644
> --- a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
> +++ b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
> @@ -4,9 +4,6 @@ Required properties:
> - compatible: "st,stm32h7-dmamux"
> - reg: Memory map for accessing module
> - #dma-cells: Should be set to <3>.
> - First parameter is request line number.
> - Second is DMA channel configuration
> - Third is Fifo threshold
> For more details about the three cells, please see
> stm32-dma.txt documentation binding file
> - dma-masters: Phandle pointing to the DMA controllers.
> @@ -53,7 +50,7 @@ dma2: dma@40020400 {
> <68>,
> <69>,
> <70>;
> - clocks = <&timer_clk>;
> + clocks = <&clk_hclk>;
> #dma-cells = <4>;
> st,mem2mem;
> resets = <&rcc 150>;
> --
> 2.7.4
>
^ permalink raw reply
* [v1,3/7] dt-bindings: stm32-mdma: Add DMA/MDMA chaining support bindings
From: Rob Herring @ 2018-09-26 20:23 UTC (permalink / raw)
To: Pierre-Yves MORDRET
Cc: Vinod Koul, Mark Rutland, Alexandre Torgue, Maxime Coquelin,
Dan Williams, devicetree, dmaengine, linux-arm-kernel,
linux-kernel
On Tue, Sep 11, 2018 at 09:26:56AM +0200, Pierre-Yves MORDRET wrote:
> This patch adds the description of the 2 properties needed to support M2M
> transfer triggered by STM32 DMA when his transfer is complete.
Same problem here.
>
> Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
> ---
> Version history:
> v1:
> * Initial
> ---
> ---
> .../devicetree/bindings/dma/stm32-mdma.txt | 22 +++++++++++++---------
> 1 file changed, 13 insertions(+), 9 deletions(-)
>
> diff --git a/Documentation/devicetree/bindings/dma/stm32-mdma.txt b/Documentation/devicetree/bindings/dma/stm32-mdma.txt
> index d18772d..1810f87 100644
> --- a/Documentation/devicetree/bindings/dma/stm32-mdma.txt
> +++ b/Documentation/devicetree/bindings/dma/stm32-mdma.txt
> @@ -10,7 +10,7 @@ Required properties:
> - interrupts: Should contain the MDMA interrupt.
> - clocks: Should contain the input clock of the DMA instance.
> - resets: Reference to a reset controller asserting the DMA controller.
> -- #dma-cells : Must be <5>. See DMA client paragraph for more details.
> +- #dma-cells : Must be <6>. See DMA client paragraph for more details.
>
> Optional properties:
> - dma-channels: Number of DMA channels supported by the controller.
> @@ -26,7 +26,7 @@ Example:
> interrupts = <122>;
> clocks = <&timer_clk>;
> resets = <&rcc 992>;
> - #dma-cells = <5>;
> + #dma-cells = <6>;
> dma-channels = <16>;
> dma-requests = <32>;
> st,ahb-addr-masks = <0x20000000>, <0x00000000>;
> @@ -35,8 +35,8 @@ Example:
> * DMA client
>
> DMA clients connected to the STM32 MDMA controller must use the format
> -described in the dma.txt file, using a five-cell specifier for each channel:
> -a phandle to the MDMA controller plus the following five integer cells:
> +described in the dma.txt file, using a six-cell specifier for each channel:
> +a phandle to the MDMA controller plus the following six integer cells:
>
> 1. The request line number
> 2. The priority level
> @@ -76,19 +76,23 @@ a phandle to the MDMA controller plus the following five integer cells:
> if no HW ack signal is used by the MDMA client
> 5. A 32bit mask specifying the value to be written to acknowledge the request
> if no HW ack signal is used by the MDMA client
> +6. A bitfield value specifying if the MDMA client wants to generate M2M
> + transfer with HW trigger (1) or not (0). This bitfield should be only
> + enabled for M2M transfer triggered by STM32 DMA client. The memory devices
> + involved in this kind of transfer are SRAM and DDR.
>
> Example:
>
> i2c4: i2c@5c002000 {
> compatible = "st,stm32f7-i2c";
> reg = <0x5c002000 0x400>;
> - interrupts = <95>,
> - <96>;
> - clocks = <&timer_clk>;
> + interrupts = <GIC_SPI 95 IRQ_TYPE_NONE>,
> + <GIC_SPI 96 IRQ_TYPE_NONE>;
> + clocks = <&clk_hsi>;
> #address-cells = <1>;
> #size-cells = <0>;
> - dmas = <&mdma1 36 0x0 0x40008 0x0 0x0>,
> - <&mdma1 37 0x0 0x40002 0x0 0x0>;
> + dmas = <&mdma1 36 0x0 0x40008 0x0 0x0 0>,
> + <&mdma1 37 0x0 0x40002 0x0 0x0 0>;
> dma-names = "rx", "tx";
> status = "disabled";
> };
> --
> 2.7.4
>
^ permalink raw reply
* dmaengine: ioat: fix prototype of ioat_enumerate_channels
From: Rami Rosen @ 2018-09-27 4:02 UTC (permalink / raw)
To: dmaengine; +Cc: dave.jiang, dan.j.williams, Rami Rosen
This patch changes the return type of the static method
ioat_init_channel() to be void. There is no need for
this method to retrun any value, and the return
value of this static method is not used anyhow.
(Setting dma->chancnt is enough).
Signed-off-by: Rami Rosen <ramirose@gmail.com>
---
drivers/dma/ioat/init.c | 7 +++----
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/drivers/dma/ioat/init.c b/drivers/dma/ioat/init.c
index 4fa4c06c9edb..c5aff6fa5ba2 100644
--- a/drivers/dma/ioat/init.c
+++ b/drivers/dma/ioat/init.c
@@ -129,7 +129,7 @@ static void
ioat_init_channel(struct ioatdma_device *ioat_dma,
struct ioatdma_chan *ioat_chan, int idx);
static void ioat_intr_quirk(struct ioatdma_device *ioat_dma);
-static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma);
+static void ioat_enumerate_channels(struct ioatdma_device *ioat_dma);
static int ioat3_dma_self_test(struct ioatdma_device *ioat_dma);
static int ioat_dca_enabled = 1;
@@ -575,7 +575,7 @@ static void ioat_dma_remove(struct ioatdma_device *ioat_dma)
* ioat_enumerate_channels - find and initialize the device's channels
* @ioat_dma: the ioat dma device to be enumerated
*/
-static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
+static void ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
{
struct ioatdma_chan *ioat_chan;
struct device *dev = &ioat_dma->pdev->dev;
@@ -594,7 +594,7 @@ static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
xfercap_log = readb(ioat_dma->reg_base + IOAT_XFERCAP_OFFSET);
xfercap_log &= 0x1f; /* bits [4:0] valid */
if (xfercap_log == 0)
- return 0;
+ return;
dev_dbg(dev, "%s: xfercap = %d\n", __func__, 1 << xfercap_log);
for (i = 0; i < dma->chancnt; i++) {
@@ -611,7 +611,6 @@ static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
}
}
dma->chancnt = i;
- return i;
}
/**
^ permalink raw reply related
* dt-bindings: dmaengine: usb-dmac: Add binding for r8a7744
From: Biju Das @ 2018-09-27 12:46 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland
Cc: Biju Das, dmaengine, devicetree, Simon Horman, Geert Uytterhoeven,
Chris Paterson, Fabrizio Castro, linux-renesas-soc
This patch adds binding for r8a7744 (RZ/G1N).
Signed-off-by: Biju Das <biju.das@bp.renesas.com>
Reviewed-by: Chris Paterson <Chris.Paterson2@renesas.com>
---
This patch is tested against linux-next next-20180927
---
Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt | 1 +
1 file changed, 1 insertion(+)
diff --git a/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt b/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt
index 482e543..1743017 100644
--- a/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt
+++ b/Documentation/devicetree/bindings/dma/renesas,usb-dmac.txt
@@ -4,6 +4,7 @@ Required Properties:
-compatible: "renesas,<soctype>-usb-dmac", "renesas,usb-dmac" as fallback.
Examples with soctypes are:
- "renesas,r8a7743-usb-dmac" (RZ/G1M)
+ - "renesas,r8a7744-usb-dmac" (RZ/G1N)
- "renesas,r8a7745-usb-dmac" (RZ/G1E)
- "renesas,r8a7790-usb-dmac" (R-Car H2)
- "renesas,r8a7791-usb-dmac" (R-Car M2-W)
^ permalink raw reply related
* dmaengine: ioat: fix prototype of ioat_enumerate_channels
From: Dave Jiang @ 2018-09-27 15:41 UTC (permalink / raw)
To: Rami Rosen, dmaengine; +Cc: dan.j.williams
On 09/26/2018 09:02 PM, Rami Rosen wrote:
> This patch changes the return type of the static method
> ioat_init_channel() to be void. There is no need for
> this method to retrun any value, and the return
> value of this static method is not used anyhow.
> (Setting dma->chancnt is enough).
>
> Signed-off-by: Rami Rosen <ramirose@gmail.com>
Acked-by: Dave Jiang <dave.jiang@intel.com>
> ---
> drivers/dma/ioat/init.c | 7 +++----
> 1 file changed, 3 insertions(+), 4 deletions(-)
>
> diff --git a/drivers/dma/ioat/init.c b/drivers/dma/ioat/init.c
> index 4fa4c06c9edb..c5aff6fa5ba2 100644
> --- a/drivers/dma/ioat/init.c
> +++ b/drivers/dma/ioat/init.c
> @@ -129,7 +129,7 @@ static void
> ioat_init_channel(struct ioatdma_device *ioat_dma,
> struct ioatdma_chan *ioat_chan, int idx);
> static void ioat_intr_quirk(struct ioatdma_device *ioat_dma);
> -static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma);
> +static void ioat_enumerate_channels(struct ioatdma_device *ioat_dma);
> static int ioat3_dma_self_test(struct ioatdma_device *ioat_dma);
>
> static int ioat_dca_enabled = 1;
> @@ -575,7 +575,7 @@ static void ioat_dma_remove(struct ioatdma_device *ioat_dma)
> * ioat_enumerate_channels - find and initialize the device's channels
> * @ioat_dma: the ioat dma device to be enumerated
> */
> -static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
> +static void ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
> {
> struct ioatdma_chan *ioat_chan;
> struct device *dev = &ioat_dma->pdev->dev;
> @@ -594,7 +594,7 @@ static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
> xfercap_log = readb(ioat_dma->reg_base + IOAT_XFERCAP_OFFSET);
> xfercap_log &= 0x1f; /* bits [4:0] valid */
> if (xfercap_log == 0)
> - return 0;
> + return;
> dev_dbg(dev, "%s: xfercap = %d\n", __func__, 1 << xfercap_log);
>
> for (i = 0; i < dma->chancnt; i++) {
> @@ -611,7 +611,6 @@ static int ioat_enumerate_channels(struct ioatdma_device *ioat_dma)
> }
> }
> dma->chancnt = i;
> - return i;
> }
>
> /**
>
^ permalink raw reply
* [v5,4/7] dmaengine: xilinx_dma: program hardware supported buffer length
From: Andrea Merello @ 2018-09-28 6:53 UTC (permalink / raw)
To: Vinod
Cc: dan.j.williams, michal.simek, appana.durga.rao, dmaengine,
linux-arm-kernel, linux-kernel, Rob Herring, Mark Rutland,
devicetree, Radhey Shyam Pandey
On Tue, Sep 18, 2018 at 6:25 PM Vinod <vkoul@kernel.org> wrote:
>
> On 07-09-18, 08:24, Andrea Merello wrote:
> > From: Radhey Shyam Pandey <radhey.shyam.pandey@xilinx.com>
> >
> > AXI-DMA IP supports configurable (c_sg_length_width) buffer length
> > register width, hence read buffer length (xlnx,sg-length-width) DT
> > property and ensure that driver doesn't program buffer length
> > exceeding the supported limit. For VDMA and CDMA there is no change.
> >
> > Cc: Rob Herring <robh+dt@kernel.org>
> > Cc: Mark Rutland <mark.rutland@arm.com>
> > Cc: devicetree@vger.kernel.org
> > Signed-off-by: Radhey Shyam Pandey <radhey.shyam.pandey@xilinx.com>
> > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > Signed-off-by: Andrea Merello <andrea.merello@gmail.com> [rebase, reword]
> > ---
> > Changes in v2:
> > - drop original patch and replace with the one in Xilinx tree
> > Changes in v3:
> > - cc DT maintainers/ML
> > Changes in v4:
> > - upper bound for the property should be 26, not 23
> > - add warn for width > 23 as per xilinx original patch
> > - rework due to changes introduced in 1/6
> > Changes in v5:
> > None
> > ---
> > drivers/dma/xilinx/xilinx_dma.c | 36 +++++++++++++++++++++++++--------
> > 1 file changed, 28 insertions(+), 8 deletions(-)
> >
> > diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
> > index aaa6de8a70e4..b17f24e4ec35 100644
> > --- a/drivers/dma/xilinx/xilinx_dma.c
> > +++ b/drivers/dma/xilinx/xilinx_dma.c
> > @@ -158,7 +158,9 @@
> > #define XILINX_DMA_REG_BTT 0x28
> >
> > /* AXI DMA Specific Masks/Bit fields */
> > -#define XILINX_DMA_MAX_TRANS_LEN GENMASK(22, 0)
> > +#define XILINX_DMA_MAX_TRANS_LEN_MIN 8
> > +#define XILINX_DMA_MAX_TRANS_LEN_MAX 23
> > +#define XILINX_DMA_V2_MAX_TRANS_LEN_MAX 26
> > #define XILINX_DMA_CR_COALESCE_MAX GENMASK(23, 16)
> > #define XILINX_DMA_CR_CYCLIC_BD_EN_MASK BIT(4)
> > #define XILINX_DMA_CR_COALESCE_SHIFT 16
> > @@ -418,6 +420,7 @@ struct xilinx_dma_config {
> > * @rxs_clk: DMA s2mm stream clock
> > * @nr_channels: Number of channels DMA device supports
> > * @chan_id: DMA channel identifier
> > + * @max_buffer_len: Max buffer length
> > */
> > struct xilinx_dma_device {
> > void __iomem *regs;
> > @@ -437,6 +440,7 @@ struct xilinx_dma_device {
> > struct clk *rxs_clk;
> > u32 nr_channels;
> > u32 chan_id;
> > + u32 max_buffer_len;
> > };
> >
> > /* Macros */
> > @@ -964,7 +968,7 @@ static int xilinx_dma_calc_copysize(struct xilinx_dma_chan *chan,
> > int size, int done)
> > {
> > size_t copy = min_t(size_t, size - done,
> > - XILINX_DMA_MAX_TRANS_LEN);
> > + chan->xdev->max_buffer_len);
>
> hmm why not add max_buffer_len in patch 1 again, and then use default
> len as XILINX_DMA_MAX_TRANS_LEN and add multiple lengths here :)
Sorry, I'm not getting your point. Could you please elaborate the "add
multiple lengths here" thing ?
> -
> ~Vinod
^ permalink raw reply
* [v5,2/7] dmaengine: xilinx_dma: in axidma slave_sg and dma_cyclic mode align split descriptors
From: Andrea Merello @ 2018-09-28 7:11 UTC (permalink / raw)
To: Vinod
Cc: dan.j.williams, michal.simek, appana.durga.rao, dmaengine,
linux-arm-kernel, linux-kernel, Rob Herring, Mark Rutland,
devicetree, Radhey Shyam Pandey
On Tue, Sep 18, 2018 at 6:21 PM Vinod <vkoul@kernel.org> wrote:
>
> On 07-09-18, 08:24, Andrea Merello wrote:
> > Whenever a single or cyclic transaction is prepared, the driver
> > could eventually split it over several SG descriptors in order
> > to deal with the HW maximum transfer length.
> >
> > This could end up in DMA operations starting from a misaligned
> > address. This seems fatal for the HW if DRE (Data Realignment Engine)
> > is not enabled.
> >
> > This patch eventually adjusts the transfer size in order to make sure
> > all operations start from an aligned address.
> >
> > Cc: Radhey Shyam Pandey <radhey.shyam.pandey@xilinx.com>
> > Signed-off-by: Andrea Merello <andrea.merello@gmail.com>
> > Reviewed-by: Radhey Shyam Pandey <radhey.shyam.pandey@xilinx.com>
> > ---
> > Changes in v2:
> > - don't introduce copy_mask field, rather rely on already-esistent
> > copy_align field. Suggested by Radhey Shyam Pandey
> > - reword title
> > Changes in v3:
> > - fix bug introduced in v2: wrong copy size when DRE is enabled
> > - use implementation suggested by Radhey Shyam Pandey
> > Changes in v4:
> > - rework on the top of 1/6
> > Changes in v5:
> > - fix typo in commit title
> > - add hint about "DRE" meaning in commit message
> > ---
> > drivers/dma/xilinx/xilinx_dma.c | 22 ++++++++++++++++++----
> > 1 file changed, 18 insertions(+), 4 deletions(-)
> >
> > diff --git a/drivers/dma/xilinx/xilinx_dma.c b/drivers/dma/xilinx/xilinx_dma.c
> > index a3aaa0e34cc7..aaa6de8a70e4 100644
> > --- a/drivers/dma/xilinx/xilinx_dma.c
> > +++ b/drivers/dma/xilinx/xilinx_dma.c
> > @@ -954,15 +954,28 @@ static int xilinx_dma_alloc_chan_resources(struct dma_chan *dchan)
> >
> > /**
> > * xilinx_dma_calc_copysize - Calculate the amount of data to copy
> > + * @chan: Driver specific DMA channel
> > * @size: Total data that needs to be copied
> > * @done: Amount of data that has been already copied
> > *
> > * Return: Amount of data that has to be copied
> > */
> > -static int xilinx_dma_calc_copysize(int size, int done)
> > +static int xilinx_dma_calc_copysize(struct xilinx_dma_chan *chan,
> > + int size, int done)
>
> align to preceeding line opening brace please
After applying, I'm seeing it already aligned as you requested; 4 tabs
+ 4 spaces so the 2nd line starts right under the "s" near the opened
brace..
Patch sent using git, so it should pass through without being ruined;
don't know why you see it misaligned :(
> > {
> > - return min_t(size_t, size - done,
> > + size_t copy = min_t(size_t, size - done,
> > XILINX_DMA_MAX_TRANS_LEN);
>
> so we can do this way in patch 1:
>
> size t copy;
>
> copy = min_t(size_t, size - done,
> XILINX_DMA_MAX_TRANS_LEN);
>
> return copy;
>
> and then add these here, feels like we are redoing change introduced in
> patch 1..
OK, this sounds good :)
>
> > + if ((copy + done < size) &&
> > + chan->xdev->common.copy_align) {
> > + /*
> > + * If this is not the last descriptor, make sure
> > + * the next one will be properly aligned
> > + */
> > + copy = rounddown(copy,
> > + (1 << chan->xdev->common.copy_align));
> > + }
> > + return copy;
> > }
> >
> > /**
> > @@ -1804,7 +1817,7 @@ static struct dma_async_tx_descriptor *xilinx_dma_prep_slave_sg(
> > * Calculate the maximum number of bytes to transfer,
> > * making sure it is less than the hw limit
> > */
> > - copy = xilinx_dma_calc_copysize(sg_dma_len(sg),
> > + copy = xilinx_dma_calc_copysize(chan, sg_dma_len(sg),
>
> why not keep chan in patch 1 and add only handling in patch 2, seems
> less churn to me..
Indeed this was something I was unsure about.. I ended up in feeling
better not to add introduce a function that takes an unused (yet)
argument, but I can change this of course :)
> --
> ~Vinod
^ permalink raw reply
* [v2,1/7] dt-bindings: stm32-dma: Add DMA/MDMA chaining support bindings
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
This patch adds dma bindings to support DMA/MDMA chaining transfer.
1 bit is to manage both DMA FIFO Threshold
1 bit is to manage DMA/MDMA Chaining features.
2 bits are used to specify SDRAM size to use for DMA/MDMA chaining.
The size in bytes of a certain order is given by the formula:
(2 ^ order) * PAGE_SIZE.
The order is given by those 2 bits.
For cyclic, whether chaining is chosen, any value above 1 can be set :
SRAM buffer size will rely on period size and not on this DT value.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
.../devicetree/bindings/dma/stm32-dma.txt | 27 +++++++++++++++++++++-
1 file changed, 26 insertions(+), 1 deletion(-)
diff --git a/Documentation/devicetree/bindings/dma/stm32-dma.txt b/Documentation/devicetree/bindings/dma/stm32-dma.txt
index c5f5190..2bac8c7 100644
--- a/Documentation/devicetree/bindings/dma/stm32-dma.txt
+++ b/Documentation/devicetree/bindings/dma/stm32-dma.txt
@@ -17,6 +17,12 @@ Optional properties:
- resets: Reference to a reset controller asserting the DMA controller
- st,mem2mem: boolean; if defined, it indicates that the controller supports
memory-to-memory transfer
+- dmas: A list of eight dma specifiers, one for each entry in dma-names.
+ Refer to stm32-mdma.txt for more details.
+- dma-names: should contain "ch0", "ch1", "ch2", "ch3", "ch4", "ch5", "ch6" and
+ "ch7" and represents each STM32 DMA channel connected to a STM32 MDMA one.
+- memory-region : phandle to a node describing memory to be used for
+ M2M intermediate transfer between DMA and MDMA.
Example:
@@ -36,6 +42,16 @@ Example:
st,mem2mem;
resets = <&rcc 150>;
dma-requests = <8>;
+ dmas = <&mdma1 8 0x10 0x1200000a 0x40026408 0x00000020 1>,
+ <&mdma1 9 0x10 0x1200000a 0x40026408 0x00000800 1>,
+ <&mdma1 10 0x10 0x1200000a 0x40026408 0x00200000 1>,
+ <&mdma1 11 0x10 0x1200000a 0x40026408 0x08000000 1>,
+ <&mdma1 12 0x10 0x1200000a 0x4002640C 0x00000020 1>,
+ <&mdma1 13 0x10 0x1200000a 0x4002640C 0x00000800 1>,
+ <&mdma1 14 0x10 0x1200000a 0x4002640C 0x00200000 1>,
+ <&mdma1 15 0x10 0x1200000a 0x4002640C 0x08000000 1>;
+ dma-names = "ch0", "ch1", "ch2", "ch3", "ch4", "ch5", "ch6", "ch7";
+ memory-region = <&sram_dmapool>;
};
* DMA client
@@ -68,7 +84,16 @@ channel: a phandle to the DMA controller plus the following four integer cells:
0x1: 1/2 full FIFO
0x2: 3/4 full FIFO
0x3: full FIFO
-
+ -bit 2: Intermediate M2M transfer from/to DDR to/from SRAM throughout MDMA
+ 0: MDMA not used to generate an intermediate M2M transfer
+ 1: MDMA used to generate an intermediate M2M transfer.
+ -bit 3-4: indicated SRAM Buffer size in (2^order)*PAGE_SIZE.
+ PAGE_SIZE is given by Linux at 4KiB: include/asm-generic/page.h.
+ Order is given by those 2 bits starting at 0.
+ Valid only whether Intermediate M2M transfer is set.
+ For cyclic, whether Intermediate M2M transfer is chosen, any value can
+ be set: SRAM buffer size will rely on period size and not on this DT
+ value.
Example:
^ permalink raw reply related
* [v2,2/7] dt-bindings: stm32-dmamux: Add one cell to support DMA/MDMA chain
From: Pierre Yves MORDRET @ 2018-09-28 8:36 UTC (permalink / raw)
To: Vinod Koul, Rob Herring, Mark Rutland, Alexandre Torgue,
Maxime Coquelin, Dan Williams, devicetree, dmaengine,
linux-arm-kernel, linux-kernel
Cc: Pierre-Yves MORDRET
From: M'boumba Cedric Madianga <cedric.madianga@gmail.com>
Add one cell to support DMA/MDMA chaining.
Signed-off-by: Pierre-Yves MORDRET <pierre-yves.mordret@st.com>
---
Version history:
v1:
* Initial
---
---
Documentation/devicetree/bindings/dma/stm32-dmamux.txt | 6 +++---
1 file changed, 3 insertions(+), 3 deletions(-)
diff --git a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
index 1b893b2..5e92b59 100644
--- a/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
+++ b/Documentation/devicetree/bindings/dma/stm32-dmamux.txt
@@ -4,9 +4,9 @@ Required properties:
- compatible: "st,stm32h7-dmamux"
- reg: Memory map for accessing module
- #dma-cells: Should be set to <3>.
- First parameter is request line number.
- Second is DMA channel configuration
- Third is Fifo threshold
+- First parameter is request line number.
+- Second is DMA channel configuration
+- Third is a 32bit bitfield
For more details about the three cells, please see
stm32-dma.txt documentation binding file
- dma-masters: Phandle pointing to the DMA controllers.
^ permalink raw reply related
page: next (older) | prev (newer) | latest
- recent:[subjects (threaded)|topics (new)|topics (active)]
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox