[dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver

All of lore.kernel.org
 help / color / mirror / Atom feed

* [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver
@ 2021-10-30 10:36 Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
                   ` (6 more replies)
  0 siblings, 7 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch set add hisilicon DMA driver.

Chengwen Feng (6):
  dma/hisilicon: add device probe and remove functions
  dma/hisilicon: add dmadev instances create and destroy
  dma/hisilicon: add control path functions
  dma/hisilicon: add data path functions
  dma/hisilicon: support multi-process
  devbind: add Kunpeng DMA to dmadev category

 MAINTAINERS                            |   5 +
 doc/guides/dmadevs/hisilicon.rst       |  41 ++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 925 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    | 236 +++++++
 drivers/dma/hisilicon/meson.build      |   7 +
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 usertools/dpdk-devbind.py              |   6 +-
 10 files changed, 1228 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

Add the basic device probe and remove functions and initial
documentation for new hisilicon DMA drivers. Maintainers update is also
included in this patch.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 MAINTAINERS                            |   5 ++
 doc/guides/dmadevs/hisilicon.rst       |  21 +++++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 119 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    |  24 +++++
 drivers/dma/hisilicon/meson.build      |   7 ++
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 9 files changed, 185 insertions(+)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 0e5951f8f1..1567f7b695 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1206,6 +1206,11 @@ M: Conor Walsh <conor.walsh@intel.com>
 F: drivers/dma/ioat/
 F: doc/guides/dmadevs/ioat.rst
 
+Hisilicon DMA
+M: Chengwen Feng <fengchengwen@huawei.com>
+F: drivers/dma/hisilicon
+F: doc/guides/dmadevs/hisilicon.rst
+
 
 RegEx Drivers
 -------------
diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
new file mode 100644
index 0000000000..4cbaac4204
--- /dev/null
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -0,0 +1,21 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2021 HiSilicon Limited.
+
+HISILICON Kunpeng DMA Driver
+============================
+
+Kunpeng SoC has an internal DMA unit which can be used by application to
+accelerate data copies. The DMA PF function supports multiple DMA channels.
+
+
+Supported Kunpeng SoCs
+----------------------
+
+* Kunpeng 920
+
+
+Device Setup
+-------------
+
+Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
+user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst
index 20476039a5..6b04276524 100644
--- a/doc/guides/dmadevs/index.rst
+++ b/doc/guides/dmadevs/index.rst
@@ -13,3 +13,4 @@ an application through DMA API.
 
    idxd
    ioat
+   hisilicon
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 502cc5ceb2..00a45475be 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -86,6 +86,10 @@ New Features
   driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake,
   Skylake and Broadwell. This device driver can be used through the generic dmadev API.
 
+* **Added hisilicon dmadev driver implementation.**
+  The hisilicon dmadev driver provide device drivers for the Kunpeng's DMA devices.
+  This device driver can be used through the generic dmadev API.
+
 * **Added support to get all MAC addresses of a device.**
 
   Added ``rte_eth_macaddrs_get`` to allow user to retrieve all Ethernet
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
new file mode 100644
index 0000000000..e6fb8a0fc8
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 HiSilicon Limited
+ */
+
+#include <inttypes.h>
+#include <string.h>
+
+#include <rte_bus_pci.h>
+#include <rte_eal.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <rte_dmadev_pmd.h>
+
+#include "hisi_dmadev.h"
+
+RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
+#define HISI_DMA_LOG(level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s(): " fmt "\n", __func__, ##args)
+#define HISI_DMA_LOG_RAW(hw, level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s %s(): " fmt "\n", (hw)->data->dev_name, \
+		__func__, ##args)
+#define HISI_DMA_DEBUG(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, DEBUG, fmt, ## args)
+#define HISI_DMA_INFO(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, INFO, fmt, ## args)
+#define HISI_DMA_WARN(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, WARNING, fmt, ## args)
+#define HISI_DMA_ERR(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
+
+static uint8_t
+hisi_dma_reg_layout(uint8_t revision)
+{
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		return HISI_DMA_REG_LAYOUT_HIP08;
+	else
+		return HISI_DMA_REG_LAYOUT_INVALID;
+}
+
+static void
+hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
+			     char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function);
+}
+
+static int
+hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
+			uint8_t *out_revision)
+{
+	uint8_t revision;
+	int ret;
+
+	ret = rte_pci_read_config(pci_dev, &revision, 1,
+				  HISI_DMA_PCI_REVISION_ID_REG);
+	if (ret != 1) {
+		HISI_DMA_LOG(ERR, "%s read PCI revision failed!", name);
+		return -EINVAL;
+	}
+	if (hisi_dma_reg_layout(revision) == HISI_DMA_REG_LAYOUT_INVALID) {
+		HISI_DMA_LOG(ERR, "%s revision: 0x%x not supported!",
+			     name, revision);
+		return -EINVAL;
+	}
+
+	*out_revision = revision;
+	return 0;
+}
+
+static int
+hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
+	       struct rte_pci_device *pci_dev)
+{
+	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
+	uint8_t revision;
+	int ret;
+
+	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
+
+	if (pci_dev->mem_resource[2].addr == NULL) {
+		HISI_DMA_LOG(ERR, "%s BAR2 is NULL!\n", name);
+		return -ENODEV;
+	}
+
+	ret = hisi_dma_check_revision(pci_dev, name, &revision);
+	if (ret)
+		return ret;
+	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
+
+	return ret;
+}
+
+static int
+hisi_dma_remove(struct rte_pci_device *pci_dev)
+{
+	RTE_SET_USED(pci_dev);
+	return 0;
+}
+
+static const struct rte_pci_id pci_id_hisi_dma_map[] = {
+	{ RTE_PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HISI_DMA_DEVICE_ID) },
+	{ .vendor_id = 0, }, /* sentinel */
+};
+
+static struct rte_pci_driver hisi_dma_pmd_drv = {
+	.id_table  = pci_id_hisi_dma_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe     = hisi_dma_probe,
+	.remove    = hisi_dma_remove,
+};
+
+RTE_PMD_REGISTER_PCI(dma_hisilicon, hisi_dma_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(dma_hisilicon, pci_id_hisi_dma_map);
+RTE_PMD_REGISTER_KMOD_DEP(dma_hisilicon, "vfio-pci");
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
new file mode 100644
index 0000000000..114b9dcb5b
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 HiSilicon Limited
+ */
+
+#ifndef HISI_DMADEV_H
+#define HISI_DMADEV_H
+
+#define PCI_VENDOR_ID_HUAWEI			0x19e5
+#define HISI_DMA_DEVICE_ID			0xA122
+#define HISI_DMA_PCI_REVISION_ID_REG		0x08
+#define HISI_DMA_REVISION_HIP08B		0x21
+
+/**
+ * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
+ * they have the same pci device id but with different pci revision.
+ * Unfortunately, they have different register layouts, so the layout
+ * enumerations are defined.
+ */
+enum {
+	HISI_DMA_REG_LAYOUT_INVALID = 0,
+	HISI_DMA_REG_LAYOUT_HIP08
+};
+
+#endif /* HISI_DMADEV_H */
diff --git a/drivers/dma/hisilicon/meson.build b/drivers/dma/hisilicon/meson.build
new file mode 100644
index 0000000000..9e31b47141
--- /dev/null
+++ b/drivers/dma/hisilicon/meson.build
@@ -0,0 +1,7 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021 HiSilicon Limited
+
+deps += ['bus_pci', 'dmadev']
+sources = files(
+        'hisi_dmadev.c',
+)
diff --git a/drivers/dma/hisilicon/version.map b/drivers/dma/hisilicon/version.map
new file mode 100644
index 0000000000..c2e0723b4c
--- /dev/null
+++ b/drivers/dma/hisilicon/version.map
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build
index a69418ce9b..6d6c054f51 100644
--- a/drivers/dma/meson.build
+++ b/drivers/dma/meson.build
@@ -4,6 +4,7 @@
 drivers = [
         'idxd',
         'ioat',
+        'hisilicon',
         'skeleton',
 ]
 std_deps = ['dmadev']
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions Chengwen Feng
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add dmadev instances create during the PCI probe, and
destroy them during the PCI remove. Internal structures and HW
definitions was also included.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 ++
 drivers/dma/hisilicon/hisi_dmadev.c | 212 +++++++++++++++++++++++++++-
 drivers/dma/hisilicon/hisi_dmadev.h |  97 +++++++++++++
 3 files changed, 318 insertions(+), 1 deletion(-)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 4cbaac4204..65138a8365 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -19,3 +19,13 @@ Device Setup
 
 Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
 user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once probed successfully, the device will appear as four ``dmadev`` which can be
+accessed using API from the ``rte_dmadev`` library.
+
+The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
+will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
+2nd ``dmadev`` name is "7b:00.0-ch1".
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index e6fb8a0fc8..b8369e7e71 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -6,7 +6,9 @@
 #include <string.h>
 
 #include <rte_bus_pci.h>
+#include <rte_cycles.h>
 #include <rte_eal.h>
+#include <rte_io.h>
 #include <rte_log.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
@@ -30,6 +32,141 @@ RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
 #define HISI_DMA_ERR(hw, fmt, args...) \
 		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
 
+static uint32_t
+hisi_dma_queue_base(struct hisi_dma_dev *hw)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08)
+		return HISI_DMA_HIP08_QUEUE_BASE;
+	else
+		return 0;
+}
+
+static void
+hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
+{
+	rte_write32(rte_cpu_to_le_32(val),
+		    (volatile void *)((char *)base + off));
+}
+
+static void
+hisi_dma_write_dev(struct hisi_dma_dev *hw, uint32_t off, uint32_t val)
+{
+	hisi_dma_write_reg(hw->io_base, off, val);
+}
+
+static void
+hisi_dma_write_queue(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t val)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	hisi_dma_write_dev(hw, off, val);
+}
+
+static uint32_t
+hisi_dma_read_reg(void *base, uint32_t off)
+{
+	uint32_t val = rte_read32((volatile void *)((char *)base + off));
+	return rte_le_to_cpu_32(val);
+}
+
+static uint32_t
+hisi_dma_read_dev(struct hisi_dma_dev *hw, uint32_t off)
+{
+	return hisi_dma_read_reg(hw->io_base, off);
+}
+
+static uint32_t
+hisi_dma_read_queue(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return hisi_dma_read_dev(hw, off);
+}
+
+static void
+hisi_dma_update_bit(struct hisi_dma_dev *hw, uint32_t off, uint32_t pos,
+		    bool set)
+{
+	uint32_t tmp = hisi_dma_read_dev(hw, off);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_dev(hw, off, tmp);
+}
+
+static void
+hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
+			  bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
+#define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
+	uint32_t timeout = 0; \
+	while (timeout++ <= (timeout_us)) { \
+		(val) = hisi_dma_read_queue(hw, HISI_DMA_QUEUE_FSM_REG); \
+		if (cond) \
+			break; \
+		rte_delay_us(sleep_us); \
+	} \
+	(cond) ? 0 : -ETIME; \
+})
+
+static int
+hisi_dma_reset_hw(struct hisi_dma_dev *hw)
+{
+#define POLL_SLEEP_US	100
+#define POLL_TIMEOUT_US	10000
+
+	uint32_t tmp;
+	int ret;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, true);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) != HISI_DMA_STATE_RUN,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "disable dma timeout!");
+		return ret;
+	}
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL1_REG,
+				  HISI_DMA_QUEUE_CTRL1_RESET_B, true);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) == HISI_DMA_STATE_IDLE,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "reset dma timeout!");
+		return ret;
+	}
+
+	return 0;
+}
+
+static void
+hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
+{
+	struct hisi_dma_dev hw;
+
+	memset(&hw, 0, sizeof(hw));
+	hw.io_base = pci_bar;
+
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		hisi_dma_update_bit(&hw, HISI_DMA_HIP08_MODE_REG,
+				    HISI_DMA_HIP08_MODE_SEL_B, true);
+}
+
 static uint8_t
 hisi_dma_reg_layout(uint8_t revision)
 {
@@ -49,6 +186,57 @@ hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function);
 }
 
+static void
+hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
+		      uint8_t queue_id, char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x-ch%u",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function, queue_id);
+}
+
+static int
+hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
+		uint8_t revision)
+{
+#define REG_PCI_BAR_INDEX	2
+
+	char name[RTE_DEV_NAME_MAX_LEN];
+	struct rte_dma_dev *dev;
+	struct hisi_dma_dev *hw;
+	int ret;
+
+	hisi_dma_gen_dev_name(pci_dev, queue_id, name, sizeof(name));
+	dev = rte_dma_pmd_allocate(name, pci_dev->device.numa_node,
+				   sizeof(*hw));
+	if (dev == NULL) {
+		HISI_DMA_LOG(ERR, "%s allocate dmadev fail!", name);
+		return -EINVAL;
+	}
+
+	dev->device = &pci_dev->device;
+
+	hw = dev->data->dev_private;
+	hw->data = dev->data;
+	hw->revision = revision;
+	hw->reg_layout = hisi_dma_reg_layout(revision);
+	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
+	hw->queue_id = queue_id;
+
+	ret = hisi_dma_reset_hw(hw);
+	if (ret) {
+		HISI_DMA_LOG(ERR, "%s init device fail!", name);
+		(void)rte_dma_pmd_release(name);
+		return -EIO;
+	}
+
+	dev->state = RTE_DMA_DEV_READY;
+	HISI_DMA_LOG(DEBUG, "%s create dmadev success!", name);
+
+	return 0;
+}
+
 static int
 hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
 			uint8_t *out_revision)
@@ -78,6 +266,7 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 {
 	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
 	uint8_t revision;
+	uint8_t i;
 	int ret;
 
 	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
@@ -92,13 +281,34 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
+	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		ret = hisi_dma_create(pci_dev, i, revision);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s create dmadev %u failed!",
+				     name, i);
+			break;
+		}
+	}
+
 	return ret;
 }
 
 static int
 hisi_dma_remove(struct rte_pci_device *pci_dev)
 {
-	RTE_SET_USED(pci_dev);
+	char name[RTE_DEV_NAME_MAX_LEN];
+	uint8_t i;
+	int ret;
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		hisi_dma_gen_dev_name(pci_dev, i, name, sizeof(name));
+		ret = rte_dma_pmd_release(name);
+		if (ret)
+			return ret;
+	}
+
 	return 0;
 }
 
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 114b9dcb5b..50aaa38b72 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -5,11 +5,24 @@
 #ifndef HISI_DMADEV_H
 #define HISI_DMADEV_H
 
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#define BIT(x)	(1ul << (x))
+#define BITS_PER_LONG	(__SIZEOF_LONG__ * 8)
+#define GENMASK(h, l) \
+		(((~0UL) << (l)) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
+#define BF_SHF(x) (__builtin_ffsll(x) - 1)
+#define FIELD_GET(mask, reg) \
+		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
+#define HISI_DMA_MAX_HW_QUEUES			4
+
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
  * they have the same pci device id but with different pci revision.
@@ -21,4 +34,88 @@ enum {
 	HISI_DMA_REG_LAYOUT_HIP08
 };
 
+/**
+ * Hardware PCI bar register MAP:
+ *
+ *     --------------
+ *     | Misc-reg-0 |
+ *     |            |
+ *     --------------   -> Queue base
+ *     |            |
+ *     | Queue-0    |
+ *     |            |
+ *     --------------   ---
+ *     |            |    ^
+ *     | Queue-1    |   Queue region
+ *     |            |    v
+ *     --------------   ---
+ *     | ...        |
+ *     | Queue-x    |
+ *     | ...        |
+ *     --------------
+ *     | Misc-reg-1 |
+ *     --------------
+ *
+ * As described above, a single queue register is continuous and occupies the
+ * length of queue-region. The global offset for a single queue register is
+ * calculated by:
+ *     offset = queue-base + (queue-id * queue-region) + reg-offset-in-region.
+ *
+ * The first part of queue region is basically the same for HIP08 and later chip
+ * register layouts, therefore, HISI_QUEUE_* registers are defined for it.
+ */
+#define HISI_DMA_QUEUE_SQ_BASE_L_REG		0x0
+#define HISI_DMA_QUEUE_SQ_BASE_H_REG		0x4
+#define HISI_DMA_QUEUE_SQ_DEPTH_REG		0x8
+#define HISI_DMA_QUEUE_SQ_TAIL_REG		0xC
+#define HISI_DMA_QUEUE_CQ_BASE_L_REG		0x10
+#define HISI_DMA_QUEUE_CQ_BASE_H_REG		0x14
+#define HISI_DMA_QUEUE_CQ_DEPTH_REG		0x18
+#define HISI_DMA_QUEUE_CQ_HEAD_REG		0x1C
+#define HISI_DMA_QUEUE_CTRL0_REG		0x20
+#define HISI_DMA_QUEUE_CTRL0_EN_B		0
+#define HISI_DMA_QUEUE_CTRL0_PAUSE_B		4
+#define HISI_DMA_QUEUE_CTRL1_REG		0x24
+#define HISI_DMA_QUEUE_CTRL1_RESET_B		0
+#define HISI_DMA_QUEUE_FSM_REG			0x30
+#define HISI_DMA_QUEUE_FSM_STS_M		GENMASK(3, 0)
+#define HISI_DMA_QUEUE_INT_STATUS_REG		0x40
+#define HISI_DMA_QUEUE_ERR_INT_NUM0_REG		0x84
+#define HISI_DMA_QUEUE_ERR_INT_NUM1_REG		0x88
+#define HISI_DMA_QUEUE_ERR_INT_NUM2_REG		0x8C
+#define HISI_DMA_QUEUE_REGION_SIZE		0x100
+
+/**
+ * HiSilicon IP08 DMA register and field define:
+ */
+#define HISI_DMA_HIP08_QUEUE_BASE			0x0
+#define HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B		2
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_REG		0x44
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_M			GENMASK(14, 0)
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG		0x90
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG		0x94
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG		0x98
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG		0x48
+#define HISI_DMA_HIP08_MODE_REG				0x217C
+#define HISI_DMA_HIP08_MODE_SEL_B			0
+#define HISI_DMA_HIP08_DUMP_START_REG			0x2000
+#define HISI_DMA_HIP08_DUMP_END_REG			0x2280
+
+/**
+ * In fact, there are multiple states, but it need to pay attention to
+ * the following two states for the driver:
+ */
+enum {
+	HISI_DMA_STATE_IDLE = 0,
+	HISI_DMA_STATE_RUN,
+};
+
+struct hisi_dma_dev {
+	struct rte_dma_dev_data *data;
+	uint8_t revision; /**< PCI revision. */
+	uint8_t reg_layout; /**< hardware register layout. */
+	void *io_base;
+	uint8_t queue_id; /**< hardware DMA queue index. */
+};
+
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data " Chengwen Feng
                   ` (3 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add control path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 +
 drivers/dma/hisilicon/hisi_dmadev.c | 385 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  99 +++++++
 3 files changed, 494 insertions(+)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 65138a8365..24bae86bdc 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -29,3 +29,13 @@ accessed using API from the ``rte_dmadev`` library.
 The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
 will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
 2nd ``dmadev`` name is "7b:00.0-ch1".
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Kunpeng DMA configuration requirements:
+
+* ``ring_size`` must be a power of two, between 32 and 8192.
+* Only one ``vchan`` is supported per ``dmadev``.
+* Silent mode is not supported.
+* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM``.
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index b8369e7e71..bcdcf4de4b 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -10,6 +10,8 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
 
@@ -41,6 +43,14 @@ hisi_dma_queue_base(struct hisi_dma_dev *hw)
 		return 0;
 }
 
+static volatile void *
+hisi_dma_queue_regaddr(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return (volatile void *)((char *)hw->io_base + off);
+}
+
 static void
 hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
 {
@@ -103,6 +113,15 @@ hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
 	hisi_dma_write_queue(hw, qoff, tmp);
 }
 
+static void
+hisi_dma_update_queue_mbit(struct hisi_dma_dev *hw, uint32_t qoff,
+			   uint32_t mask, bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
 #define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
 	uint32_t timeout = 0; \
 	while (timeout++ <= (timeout_us)) { \
@@ -154,6 +173,45 @@ hisi_dma_reset_hw(struct hisi_dma_dev *hw)
 	return 0;
 }
 
+static void
+hisi_dma_init_hw(struct hisi_dma_dev *hw)
+{
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_L_REG,
+			     lower_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_H_REG,
+			     upper_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_L_REG,
+			     lower_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_H_REG,
+			     upper_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_DEPTH_REG,
+			     hw->sq_depth_mask);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_DEPTH_REG, hw->cq_depth - 1);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM0_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM1_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM2_REG, 0);
+
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG,
+				     0);
+		hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B, false);
+		hisi_dma_update_queue_mbit(hw, HISI_DMA_QUEUE_INT_STATUS_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+		hisi_dma_update_queue_mbit(hw,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+	}
+}
+
 static void
 hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
 {
@@ -176,6 +234,301 @@ hisi_dma_reg_layout(uint8_t revision)
 		return HISI_DMA_REG_LAYOUT_INVALID;
 }
 
+static void
+hisi_dma_zero_iomem(struct hisi_dma_dev *hw)
+{
+	memset(hw->iomz->addr, 0, hw->iomz_sz);
+}
+
+static int
+hisi_dma_alloc_iomem(struct hisi_dma_dev *hw, uint16_t ring_size,
+		     const char *dev_name)
+{
+	uint32_t sq_size = sizeof(struct hisi_dma_sqe) * ring_size;
+	uint32_t cq_size = sizeof(struct hisi_dma_cqe) *
+			   (ring_size + HISI_DMA_CQ_RESERVED);
+	uint32_t status_size = sizeof(uint16_t) * ring_size;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	const struct rte_memzone *iomz;
+	uint32_t total_size;
+
+	sq_size = RTE_CACHE_LINE_ROUNDUP(sq_size);
+	cq_size = RTE_CACHE_LINE_ROUNDUP(cq_size);
+	status_size = RTE_CACHE_LINE_ROUNDUP(status_size);
+	total_size = sq_size + cq_size + status_size;
+
+	(void)snprintf(mz_name, sizeof(mz_name), "hisi_dma:%s", dev_name);
+	iomz = rte_memzone_reserve(mz_name, total_size, hw->data->numa_node,
+				   RTE_MEMZONE_IOVA_CONTIG);
+	if (iomz == NULL) {
+		HISI_DMA_ERR(hw, "malloc %s iomem fail!", mz_name);
+		return -ENOMEM;
+	}
+
+	hw->iomz = iomz;
+	hw->iomz_sz = total_size;
+	hw->sqe = iomz->addr;
+	hw->cqe = (void *)((char *)iomz->addr + sq_size);
+	hw->status = (void *)((char *)iomz->addr + sq_size + cq_size);
+	hw->sqe_iova = iomz->iova;
+	hw->cqe_iova = iomz->iova + sq_size;
+	hw->sq_depth_mask = ring_size - 1;
+	hw->cq_depth = ring_size + HISI_DMA_CQ_RESERVED;
+	hisi_dma_zero_iomem(hw);
+
+	return 0;
+}
+
+static void
+hisi_dma_free_iomem(struct hisi_dma_dev *hw)
+{
+	if (hw->iomz != NULL)
+		rte_memzone_free(hw->iomz);
+
+	hw->iomz = NULL;
+	hw->sqe = NULL;
+	hw->cqe = NULL;
+	hw->status = NULL;
+	hw->sqe_iova = 0;
+	hw->cqe_iova = 0;
+	hw->sq_depth_mask = 0;
+	hw->cq_depth = 0;
+}
+
+static int
+hisi_dma_info_get(const struct rte_dma_dev *dev,
+		  struct rte_dma_info *dev_info,
+		  uint32_t info_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info_sz);
+
+	dev_info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM |
+			     RTE_DMA_CAPA_OPS_COPY;
+	dev_info->max_vchans = 1;
+	dev_info->max_desc = HISI_DMA_MAX_DESC_NUM;
+	dev_info->min_desc = HISI_DMA_MIN_DESC_NUM;
+
+	return 0;
+}
+
+static int
+hisi_dma_configure(struct rte_dma_dev *dev,
+		   const struct rte_dma_conf *conf,
+		   uint32_t conf_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(conf);
+	RTE_SET_USED(conf_sz);
+	return 0;
+}
+
+static int
+hisi_dma_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan,
+		     const struct rte_dma_vchan_conf *conf,
+		     uint32_t conf_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+	int ret;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(conf_sz);
+
+	if (!rte_is_power_of_2(conf->nb_desc)) {
+		HISI_DMA_ERR(hw, "Number of desc must be power of 2!");
+		return -EINVAL;
+	}
+
+	hisi_dma_free_iomem(hw);
+	ret = hisi_dma_alloc_iomem(hw, conf->nb_desc, dev->data->dev_name);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int
+hisi_dma_start(struct rte_dma_dev *dev)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	if (hw->iomz == NULL) {
+		HISI_DMA_ERR(hw, "Vchan was not setup, start fail!\n");
+		return -EINVAL;
+	}
+
+	/* Reset the dmadev to a known state, include:
+	 *   1) zero iomem, also include status fields.
+	 *   2) init hardware register.
+	 *   3) init index values to zero.
+	 *   4) init running statistics.
+	 */
+	hisi_dma_zero_iomem(hw);
+	hisi_dma_init_hw(hw);
+	hw->ridx = 0;
+	hw->cridx = 0;
+	hw->sq_head = 0;
+	hw->sq_tail = 0;
+	hw->cq_sq_head = 0;
+	hw->cq_head = 0;
+	hw->cqs_completed = 0;
+	hw->cqe_vld = 1;
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, true);
+
+	return 0;
+}
+
+static int
+hisi_dma_stop(struct rte_dma_dev *dev)
+{
+	return hisi_dma_reset_hw(dev->data->dev_private);
+}
+
+static int
+hisi_dma_close(struct rte_dma_dev *dev)
+{
+	/* The dmadev already stopped */
+	hisi_dma_free_iomem(dev->data->dev_private);
+	return 0;
+}
+
+static int
+hisi_dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan,
+		   struct rte_dma_stats *stats,
+		   uint32_t stats_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(stats_sz);
+	stats->submitted = hw->submitted;
+	stats->completed = hw->completed;
+	stats->errors = hw->errors;
+
+	return 0;
+}
+
+static int
+hisi_dma_stats_reset(struct rte_dma_dev *dev, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	return 0;
+}
+
+static void
+hisi_dma_get_dump_range(struct hisi_dma_dev *hw, uint32_t *start, uint32_t *end)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		*start = HISI_DMA_HIP08_DUMP_START_REG;
+		*end = HISI_DMA_HIP08_DUMP_END_REG;
+	} else {
+		*start = 0;
+		*end = 0;
+	}
+}
+
+static void
+hisi_dma_dump_common(struct hisi_dma_dev *hw, FILE *f)
+{
+#define DUMP_REGNUM_PER_LINE	4
+
+	uint32_t start, end;
+	uint32_t cnt, i;
+
+	hisi_dma_get_dump_range(hw, &start, &end);
+
+	(void)fprintf(f, "    common-register:\n");
+
+	cnt = 0;
+	for (i = start; i <= end; i += sizeof(uint32_t)) {
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "      [%4x]:", i);
+		(void)fprintf(f, " 0x%08x", hisi_dma_read_dev(hw, i));
+		cnt++;
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "\n");
+	}
+	if (cnt % DUMP_REGNUM_PER_LINE)
+		(void)fprintf(f, "\n");
+}
+
+static void
+hisi_dma_dump_read_queue(struct hisi_dma_dev *hw, uint32_t qoff,
+			 char *buffer, int max_sz)
+{
+	memset(buffer, 0, max_sz);
+
+	/* Address-related registers are not printed for security reasons. */
+	if (qoff == HISI_DMA_QUEUE_SQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_SQ_BASE_H_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_H_REG) {
+		(void)snprintf(buffer, max_sz, "**********");
+		return;
+	}
+
+	(void)snprintf(buffer, max_sz, "0x%08x", hisi_dma_read_queue(hw, qoff));
+}
+
+static void
+hisi_dma_dump_queue(struct hisi_dma_dev *hw, FILE *f)
+{
+#define REG_FMT_LEN	32
+	char buf[REG_FMT_LEN] = { 0 };
+	uint32_t i;
+
+	(void)fprintf(f, "    queue-register:\n");
+	for (i = 0; i < HISI_DMA_QUEUE_REGION_SIZE; ) {
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, "      [%2x]: %s", i, buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s\n", buf);
+		i += sizeof(uint32_t);
+	}
+}
+
+static int
+hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	(void)fprintf(f,
+		"    revision: 0x%x queue_id: %u ring_size: %u\n"
+		"    ridx: %u cridx: %u\n"
+		"    sq_head: %u sq_tail: %u cq_sq_head: %u\n"
+		"    cq_head: %u cqs_completed: %u cqe_vld: %u\n"
+		"    submitted: %" PRIu64 " completed: %" PRIu64 " errors %"
+		PRIu64"\n",
+		hw->revision, hw->queue_id,
+		hw->sq_depth_mask > 0 ? hw->sq_depth_mask + 1 : 0,
+		hw->ridx, hw->cridx,
+		hw->sq_head, hw->sq_tail, hw->cq_sq_head,
+		hw->cq_head, hw->cqs_completed, hw->cqe_vld,
+		hw->submitted, hw->completed, hw->errors);
+	hisi_dma_dump_queue(hw, f);
+	hisi_dma_dump_common(hw, f);
+
+	return 0;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -196,6 +549,33 @@ hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function, queue_id);
 }
 
+/**
+ * Hardware queue state machine:
+ *
+ *   -----------  dmadev_create	  ------------------
+ *   | Unknown | ---------------> |      IDLE      |
+ *   -----------                  ------------------
+ *                                   ^          |
+ *                                   |          |dev_start
+ *                           dev_stop|          |
+ *                                   |          v
+ *                                ------------------
+ *                                |      RUN       |
+ *                                ------------------
+ *
+ */
+static const struct rte_dma_dev_ops hisi_dmadev_ops = {
+	.dev_info_get     = hisi_dma_info_get,
+	.dev_configure    = hisi_dma_configure,
+	.dev_start        = hisi_dma_start,
+	.dev_stop         = hisi_dma_stop,
+	.dev_close        = hisi_dma_close,
+	.vchan_setup      = hisi_dma_vchan_setup,
+	.stats_get        = hisi_dma_stats_get,
+	.stats_reset      = hisi_dma_stats_reset,
+	.dev_dump         = hisi_dma_dump,
+};
+
 static int
 hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 		uint8_t revision)
@@ -216,6 +596,7 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	}
 
 	dev->device = &pci_dev->device;
+	dev->dev_ops = &hisi_dmadev_ops;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
@@ -223,6 +604,10 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->reg_layout = hisi_dma_reg_layout(revision);
 	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
 	hw->queue_id = queue_id;
+	hw->sq_tail_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_SQ_TAIL_REG);
+	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
 	ret = hisi_dma_reset_hw(hw);
 	if (ret) {
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 50aaa38b72..dd0315cd31 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -16,12 +16,17 @@
 #define FIELD_GET(mask, reg) \
 		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
 
+#define lower_32_bits(x) ((uint32_t)(x))
+#define upper_32_bits(x) ((uint32_t)(((x) >> 16) >> 16))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
 #define HISI_DMA_MAX_HW_QUEUES			4
+#define HISI_DMA_MAX_DESC_NUM			8192
+#define HISI_DMA_MIN_DESC_NUM			32
 
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
@@ -110,12 +115,106 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * After scanning the CQ array, the CQ head register needs to be updated.
+ * Updating the register involves write memory barrier operations.
+ * Here use the following method to reduce WMB operations:
+ *   a) malloc more CQEs, which correspond to the macro HISI_DMA_CQ_RESERVED.
+ *   b) update the CQ head register after accumulated number of completed CQs
+ *      is greater than or equal to HISI_DMA_CQ_RESERVED.
+ */
+#define HISI_DMA_CQ_RESERVED		64
+
+struct hisi_dma_sqe {
+	uint32_t dw0;
+#define SQE_FENCE_FLAG	BIT(10)
+#define SQE_OPCODE_M2M	0x4
+	uint32_t dw1;
+	uint32_t dw2;
+	uint32_t length;
+	uint64_t src_addr;
+	uint64_t dst_addr;
+};
+
+struct hisi_dma_cqe {
+	uint64_t rsv;
+	uint64_t misc;
+#define CQE_SQ_HEAD_MASK	GENMASK(15, 0)
+#define CQE_VALID_B		BIT(48)
+#define CQE_STATUS_MASK		GENMASK(63, 49)
+};
+
 struct hisi_dma_dev {
+	struct hisi_dma_sqe *sqe;
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t *status; /* the completion status array of SQEs. */
+
+	volatile void *sq_tail_reg; /**< register address for doorbell. */
+	volatile void *cq_head_reg; /**< register address for answer CQ. */
+
+	uint16_t sq_depth_mask; /**< SQ depth - 1, the SQ depth is power of 2 */
+	uint16_t cq_depth; /* CQ depth */
+
+	uint16_t ridx; /**< ring index which will assign to the next request. */
+	/** ring index which returned by hisi_dmadev_completed APIs. */
+	uint16_t cridx;
+
+	/**
+	 * SQE array management fields:
+	 *
+	 *  -----------------------------------------------------
+	 *  | SQE0 | SQE1 | SQE2 |   ...  | SQEx | ... | SQEn-1 |
+	 *  -----------------------------------------------------
+	 *     ^             ^               ^
+	 *     |             |               |
+	 *   sq_head     cq_sq_head       sq_tail
+	 *
+	 *  sq_head: index to the oldest completed request, this filed was
+	 *           updated by hisi_dmadev_completed* APIs.
+	 *  sq_tail: index of the next new request, this field was updated by
+	 *           hisi_dmadev_copy API.
+	 *  cq_sq_head: next index of index that has been completed by hardware,
+	 *              this filed was updated by hisi_dmadev_completed* APIs.
+	 *
+	 *  [sq_head, cq_sq_head): the SQEs that hardware already completed.
+	 *  [cq_sq_head, sq_tail): the SQEs that hardware processing.
+	 */
+	uint16_t sq_head;
+	uint16_t sq_tail;
+	uint16_t cq_sq_head;
+	/**
+	 * The driver scans the CQE array, if the valid bit changes, the CQE is
+	 * considered valid.
+	 * Note: One CQE is corresponding to one or several SQEs, e.g. app
+	 *       submits two copy requests, the hardware processes the two SQEs,
+	 *       but it may write back only one CQE and the CQE's sq_head field
+	 *       indicates the index of the second copy request in the SQE
+	 *       array.
+	 */
+	uint16_t cq_head; /**< CQ index for next scans. */
+	/** accumulated number of completed CQs
+	 * @see HISI_DMA_CQ_RESERVED
+	 */
+	uint16_t cqs_completed;
+	uint8_t cqe_vld; /**< valid bit for CQE, will change for every round. */
+
+	uint64_t submitted;
+	uint64_t completed;
+	uint64_t errors;
+
+	/**
+	 * The following fields are not accessed in the I/O path, so they are
+	 * placed at the end.
+	 */
 	struct rte_dma_dev_data *data;
 	uint8_t revision; /**< PCI revision. */
 	uint8_t reg_layout; /**< hardware register layout. */
 	void *io_base;
 	uint8_t queue_id; /**< hardware DMA queue index. */
+	const struct rte_memzone *iomz;
+	uint32_t iomz_sz;
+	rte_iova_t sqe_iova;
+	rte_iova_t cqe_iova;
 };
 
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data path functions
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (2 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process Chengwen Feng
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add data path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 206 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  16 +++
 2 files changed, 222 insertions(+)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index bcdcf4de4b..d03967cae3 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -529,6 +529,206 @@ hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
 	return 0;
 }
 
+static int
+hisi_dma_copy(void *dev_private, uint16_t vchan,
+		 rte_iova_t src, rte_iova_t dst,
+		 uint32_t length, uint64_t flags)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	struct hisi_dma_sqe *sqe = &hw->sqe[hw->sq_tail];
+
+	RTE_SET_USED(vchan);
+
+	if (((hw->sq_tail + 1) & hw->sq_depth_mask) == hw->sq_head)
+		return -ENOSPC;
+
+	sqe->dw0 = rte_cpu_to_le_32(SQE_OPCODE_M2M);
+	sqe->dw1 = 0;
+	sqe->dw2 = 0;
+	sqe->length = rte_cpu_to_le_32(length);
+	sqe->src_addr = rte_cpu_to_le_64(src);
+	sqe->dst_addr = rte_cpu_to_le_64(dst);
+	hw->sq_tail = (hw->sq_tail + 1) & hw->sq_depth_mask;
+	hw->submitted++;
+
+	if (flags & RTE_DMA_OP_FLAG_FENCE)
+		sqe->dw0 |= rte_cpu_to_le_32(SQE_FENCE_FLAG);
+	if (flags & RTE_DMA_OP_FLAG_SUBMIT)
+		rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return hw->ridx++;
+}
+
+static int
+hisi_dma_submit(void *dev_private, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev_private;
+
+	RTE_SET_USED(vchan);
+	rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return 0;
+}
+
+static inline void
+hisi_dma_scan_cq(struct hisi_dma_dev *hw)
+{
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t csq_head = hw->cq_sq_head;
+	uint16_t cq_head = hw->cq_head;
+	uint16_t count = 0;
+	uint64_t misc;
+
+	while (true) {
+		cqe = &hw->cqe[cq_head];
+		misc = cqe->misc;
+		misc = rte_le_to_cpu_64(misc);
+		if (FIELD_GET(CQE_VALID_B, misc) != hw->cqe_vld)
+			break;
+
+		csq_head = FIELD_GET(CQE_SQ_HEAD_MASK, misc);
+		if (unlikely(misc & CQE_STATUS_MASK))
+			hw->status[csq_head] = FIELD_GET(CQE_STATUS_MASK,
+							 misc);
+
+		count++;
+		cq_head++;
+		if (cq_head == hw->cq_depth) {
+			hw->cqe_vld = !hw->cqe_vld;
+			cq_head = 0;
+		}
+	}
+
+	if (count == 0)
+		return;
+
+	hw->cq_head = cq_head;
+	hw->cq_sq_head = (csq_head + 1) & hw->sq_depth_mask;
+	hw->cqs_completed += count;
+	if (hw->cqs_completed >= HISI_DMA_CQ_RESERVED) {
+		rte_write32(rte_cpu_to_le_32(cq_head), hw->cq_head_reg);
+		hw->cqs_completed = 0;
+	}
+}
+
+static inline uint16_t
+hisi_dma_calc_cpls(struct hisi_dma_dev *hw, const uint16_t nb_cpls)
+{
+	uint16_t cpl_num;
+
+	if (hw->cq_sq_head >= hw->sq_head)
+		cpl_num = hw->cq_sq_head - hw->sq_head;
+	else
+		cpl_num = hw->sq_depth_mask + 1 - hw->sq_head + hw->cq_sq_head;
+
+	if (cpl_num > nb_cpls)
+		cpl_num = nb_cpls;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_completed(void *dev_private,
+		   uint16_t vchan, const uint16_t nb_cpls,
+		   uint16_t *last_idx, bool *has_error)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		if (hw->status[sq_head]) {
+			*has_error = true;
+			break;
+		}
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (i > 0) {
+		hw->cridx += i;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += i;
+
+	return i;
+}
+
+static enum rte_dma_status_code
+hisi_dma_convert_status(uint16_t status)
+{
+	switch (status) {
+	case HISI_DMA_STATUS_SUCCESS:
+		return RTE_DMA_STATUS_SUCCESSFUL;
+	case HISI_DMA_STATUS_INVALID_OPCODE:
+		return RTE_DMA_STATUS_INVALID_OPCODE;
+	case HISI_DMA_STATUS_INVALID_LENGTH:
+		return RTE_DMA_STATUS_INVALID_LENGTH;
+	case HISI_DMA_STATUS_USER_ABORT:
+		return RTE_DMA_STATUS_USER_ABORT;
+	case HISI_DMA_STATUS_REMOTE_READ_ERROR:
+	case HISI_DMA_STATUS_AXI_READ_ERROR:
+		return RTE_DMA_STATUS_BUS_READ_ERROR;
+	case HISI_DMA_STATUS_AXI_WRITE_ERROR:
+		return RTE_DMA_STATUS_BUS_WRITE_ERROR;
+	case HISI_DMA_STATUS_DATA_POISON:
+	case HISI_DMA_STATUS_REMOTE_DATA_POISION:
+		return RTE_DMA_STATUS_DATA_POISION;
+	case HISI_DMA_STATUS_SQE_READ_ERROR:
+	case HISI_DMA_STATUS_SQE_READ_POISION:
+		return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR;
+	case HISI_DMA_STATUS_LINK_DOWN_ERROR:
+		return RTE_DMA_STATUS_DEV_LINK_ERROR;
+	default:
+		return RTE_DMA_STATUS_ERROR_UNKNOWN;
+	}
+}
+
+static uint16_t
+hisi_dma_completed_status(void *dev_private,
+			  uint16_t vchan, const uint16_t nb_cpls,
+			  uint16_t *last_idx, enum rte_dma_status_code *status)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		status[i] = hisi_dma_convert_status(hw->status[sq_head]);
+		hw->errors += !!status[i];
+		hw->status[sq_head] = HISI_DMA_STATUS_SUCCESS;
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (likely(cpl_num > 0)) {
+		hw->cridx += cpl_num;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += cpl_num;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_burst_capacity(const void *dev_private, uint16_t vchan)
+{
+	const struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t sq_tail = hw->sq_tail;
+
+	RTE_SET_USED(vchan);
+
+	return (sq_tail >= sq_head) ? hw->sq_depth_mask - sq_tail + sq_head :
+				      sq_head - 1 - sq_tail;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -597,6 +797,12 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 
 	dev->device = &pci_dev->device;
 	dev->dev_ops = &hisi_dmadev_ops;
+	dev->fp_obj->dev_private = dev->data->dev_private;
+	dev->fp_obj->copy = hisi_dma_copy;
+	dev->fp_obj->submit = hisi_dma_submit;
+	dev->fp_obj->completed = hisi_dma_completed;
+	dev->fp_obj->completed_status = hisi_dma_completed_status;
+	dev->fp_obj->burst_capacity = hisi_dma_burst_capacity;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index dd0315cd31..12e209c86e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -115,6 +115,22 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * Hardware complete status define:
+ */
+#define HISI_DMA_STATUS_SUCCESS			0x0
+#define HISI_DMA_STATUS_INVALID_OPCODE		0x1
+#define HISI_DMA_STATUS_INVALID_LENGTH		0x2
+#define HISI_DMA_STATUS_USER_ABORT		0x4
+#define HISI_DMA_STATUS_REMOTE_READ_ERROR	0x10
+#define HISI_DMA_STATUS_AXI_READ_ERROR		0x20
+#define HISI_DMA_STATUS_AXI_WRITE_ERROR		0x40
+#define HISI_DMA_STATUS_DATA_POISON		0x80
+#define HISI_DMA_STATUS_SQE_READ_ERROR		0x100
+#define HISI_DMA_STATUS_SQE_READ_POISION	0x200
+#define HISI_DMA_STATUS_REMOTE_DATA_POISION	0x400
+#define HISI_DMA_STATUS_LINK_DOWN_ERROR		0x800
+
 /**
  * After scanning the CQ array, the CQ head register needs to be updated.
  * Updating the register involves write memory barrier operations.
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (3 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data " Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add multi-process support for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index d03967cae3..05066b4d0e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -392,8 +392,10 @@ hisi_dma_stop(struct rte_dma_dev *dev)
 static int
 hisi_dma_close(struct rte_dma_dev *dev)
 {
-	/* The dmadev already stopped */
-	hisi_dma_free_iomem(dev->data->dev_private);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		/* The dmadev already stopped */
+		hisi_dma_free_iomem(dev->data->dev_private);
+	}
 	return 0;
 }
 
@@ -815,11 +817,13 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
 						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
-	ret = hisi_dma_reset_hw(hw);
-	if (ret) {
-		HISI_DMA_LOG(ERR, "%s init device fail!", name);
-		(void)rte_dma_pmd_release(name);
-		return -EIO;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = hisi_dma_reset_hw(hw);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s init device fail!", name);
+			(void)rte_dma_pmd_release(name);
+			return -EIO;
+		}
 	}
 
 	dev->state = RTE_DMA_DEV_READY;
@@ -872,7 +876,8 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
-	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
 
 	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
 		ret = hisi_dma_create(pci_dev, i, revision);
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (4 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process Chengwen Feng
@ 2021-10-30 10:36 ` Chengwen Feng
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-10-30 10:36 UTC (permalink / raw)
  To: thomas; +Cc: dev

add Kunpeng DMA device ID to dmadev category.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 usertools/dpdk-devbind.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index bb00f43702..a74a68ed82 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -68,10 +68,14 @@
 intel_ntb_icx = {'Class': '06', 'Vendor': '8086', 'Device': '347e',
                  'SVendor': None, 'SDevice': None}
 
+hisilicon_dma = {'Class': '08', 'Vendor': '19e5', 'Device': 'a122',
+                 'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
-dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx]
+dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx,
+               hisilicon_dma]
 eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver
  2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
                   ` (5 preceding siblings ...)
  2021-10-30 10:36 ` [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
@ 2021-11-02 12:37 ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
                     ` (6 more replies)
  6 siblings, 7 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch set add hisilicon DMA driver.

Chengwen Feng (6):
  dma/hisilicon: add device probe and remove functions
  dma/hisilicon: add dmadev instances create and destroy
  dma/hisilicon: add control path functions
  dma/hisilicon: add data path functions
  dma/hisilicon: support multi-process
  devbind: add Kunpeng DMA to dmadev category

---
v2: fix compile error on non-Linux OS.

 MAINTAINERS                            |   5 +
 doc/guides/dmadevs/hisilicon.rst       |  41 ++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 925 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    | 236 +++++++
 drivers/dma/hisilicon/meson.build      |  19 +
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 usertools/dpdk-devbind.py              |   6 +-
 10 files changed, 1240 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

-- 
2.33.0


^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-07 17:28     ` Thomas Monjalon
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
                     ` (5 subsequent siblings)
  6 siblings, 1 reply; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

Add the basic device probe and remove functions and initial
documentation for new hisilicon DMA drivers. Maintainers update is also
included in this patch.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 MAINTAINERS                            |   5 ++
 doc/guides/dmadevs/hisilicon.rst       |  21 +++++
 doc/guides/dmadevs/index.rst           |   1 +
 doc/guides/rel_notes/release_21_11.rst |   4 +
 drivers/dma/hisilicon/hisi_dmadev.c    | 119 +++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h    |  24 +++++
 drivers/dma/hisilicon/meson.build      |  19 ++++
 drivers/dma/hisilicon/version.map      |   3 +
 drivers/dma/meson.build                |   1 +
 9 files changed, 197 insertions(+)
 create mode 100644 doc/guides/dmadevs/hisilicon.rst
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.c
 create mode 100644 drivers/dma/hisilicon/hisi_dmadev.h
 create mode 100644 drivers/dma/hisilicon/meson.build
 create mode 100644 drivers/dma/hisilicon/version.map

diff --git a/MAINTAINERS b/MAINTAINERS
index 0e5951f8f1..1567f7b695 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -1206,6 +1206,11 @@ M: Conor Walsh <conor.walsh@intel.com>
 F: drivers/dma/ioat/
 F: doc/guides/dmadevs/ioat.rst
 
+Hisilicon DMA
+M: Chengwen Feng <fengchengwen@huawei.com>
+F: drivers/dma/hisilicon
+F: doc/guides/dmadevs/hisilicon.rst
+
 
 RegEx Drivers
 -------------
diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
new file mode 100644
index 0000000000..4cbaac4204
--- /dev/null
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -0,0 +1,21 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2021 HiSilicon Limited.
+
+HISILICON Kunpeng DMA Driver
+============================
+
+Kunpeng SoC has an internal DMA unit which can be used by application to
+accelerate data copies. The DMA PF function supports multiple DMA channels.
+
+
+Supported Kunpeng SoCs
+----------------------
+
+* Kunpeng 920
+
+
+Device Setup
+-------------
+
+Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
+user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
diff --git a/doc/guides/dmadevs/index.rst b/doc/guides/dmadevs/index.rst
index 20476039a5..6b04276524 100644
--- a/doc/guides/dmadevs/index.rst
+++ b/doc/guides/dmadevs/index.rst
@@ -13,3 +13,4 @@ an application through DMA API.
 
    idxd
    ioat
+   hisilicon
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 502cc5ceb2..00a45475be 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -86,6 +86,10 @@ New Features
   driver for Intel IOAT devices such as Crystal Beach DMA (CBDMA) on Ice Lake,
   Skylake and Broadwell. This device driver can be used through the generic dmadev API.
 
+* **Added hisilicon dmadev driver implementation.**
+  The hisilicon dmadev driver provide device drivers for the Kunpeng's DMA devices.
+  This device driver can be used through the generic dmadev API.
+
 * **Added support to get all MAC addresses of a device.**
 
   Added ``rte_eth_macaddrs_get`` to allow user to retrieve all Ethernet
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
new file mode 100644
index 0000000000..e6fb8a0fc8
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -0,0 +1,119 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(C) 2021 HiSilicon Limited
+ */
+
+#include <inttypes.h>
+#include <string.h>
+
+#include <rte_bus_pci.h>
+#include <rte_eal.h>
+#include <rte_log.h>
+#include <rte_pci.h>
+#include <rte_dmadev_pmd.h>
+
+#include "hisi_dmadev.h"
+
+RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
+#define HISI_DMA_LOG(level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s(): " fmt "\n", __func__, ##args)
+#define HISI_DMA_LOG_RAW(hw, level, fmt, args...) \
+		rte_log(RTE_LOG_ ## level, hisi_dma_logtype, \
+		"%s %s(): " fmt "\n", (hw)->data->dev_name, \
+		__func__, ##args)
+#define HISI_DMA_DEBUG(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, DEBUG, fmt, ## args)
+#define HISI_DMA_INFO(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, INFO, fmt, ## args)
+#define HISI_DMA_WARN(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, WARNING, fmt, ## args)
+#define HISI_DMA_ERR(hw, fmt, args...) \
+		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
+
+static uint8_t
+hisi_dma_reg_layout(uint8_t revision)
+{
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		return HISI_DMA_REG_LAYOUT_HIP08;
+	else
+		return HISI_DMA_REG_LAYOUT_INVALID;
+}
+
+static void
+hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
+			     char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function);
+}
+
+static int
+hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
+			uint8_t *out_revision)
+{
+	uint8_t revision;
+	int ret;
+
+	ret = rte_pci_read_config(pci_dev, &revision, 1,
+				  HISI_DMA_PCI_REVISION_ID_REG);
+	if (ret != 1) {
+		HISI_DMA_LOG(ERR, "%s read PCI revision failed!", name);
+		return -EINVAL;
+	}
+	if (hisi_dma_reg_layout(revision) == HISI_DMA_REG_LAYOUT_INVALID) {
+		HISI_DMA_LOG(ERR, "%s revision: 0x%x not supported!",
+			     name, revision);
+		return -EINVAL;
+	}
+
+	*out_revision = revision;
+	return 0;
+}
+
+static int
+hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
+	       struct rte_pci_device *pci_dev)
+{
+	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
+	uint8_t revision;
+	int ret;
+
+	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
+
+	if (pci_dev->mem_resource[2].addr == NULL) {
+		HISI_DMA_LOG(ERR, "%s BAR2 is NULL!\n", name);
+		return -ENODEV;
+	}
+
+	ret = hisi_dma_check_revision(pci_dev, name, &revision);
+	if (ret)
+		return ret;
+	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
+
+	return ret;
+}
+
+static int
+hisi_dma_remove(struct rte_pci_device *pci_dev)
+{
+	RTE_SET_USED(pci_dev);
+	return 0;
+}
+
+static const struct rte_pci_id pci_id_hisi_dma_map[] = {
+	{ RTE_PCI_DEVICE(PCI_VENDOR_ID_HUAWEI, HISI_DMA_DEVICE_ID) },
+	{ .vendor_id = 0, }, /* sentinel */
+};
+
+static struct rte_pci_driver hisi_dma_pmd_drv = {
+	.id_table  = pci_id_hisi_dma_map,
+	.drv_flags = RTE_PCI_DRV_NEED_MAPPING,
+	.probe     = hisi_dma_probe,
+	.remove    = hisi_dma_remove,
+};
+
+RTE_PMD_REGISTER_PCI(dma_hisilicon, hisi_dma_pmd_drv);
+RTE_PMD_REGISTER_PCI_TABLE(dma_hisilicon, pci_id_hisi_dma_map);
+RTE_PMD_REGISTER_KMOD_DEP(dma_hisilicon, "vfio-pci");
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
new file mode 100644
index 0000000000..114b9dcb5b
--- /dev/null
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -0,0 +1,24 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2021 HiSilicon Limited
+ */
+
+#ifndef HISI_DMADEV_H
+#define HISI_DMADEV_H
+
+#define PCI_VENDOR_ID_HUAWEI			0x19e5
+#define HISI_DMA_DEVICE_ID			0xA122
+#define HISI_DMA_PCI_REVISION_ID_REG		0x08
+#define HISI_DMA_REVISION_HIP08B		0x21
+
+/**
+ * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
+ * they have the same pci device id but with different pci revision.
+ * Unfortunately, they have different register layouts, so the layout
+ * enumerations are defined.
+ */
+enum {
+	HISI_DMA_REG_LAYOUT_INVALID = 0,
+	HISI_DMA_REG_LAYOUT_HIP08
+};
+
+#endif /* HISI_DMADEV_H */
diff --git a/drivers/dma/hisilicon/meson.build b/drivers/dma/hisilicon/meson.build
new file mode 100644
index 0000000000..0786772442
--- /dev/null
+++ b/drivers/dma/hisilicon/meson.build
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2021 HiSilicon Limited
+
+if not is_linux
+    build = false
+    reason = 'only supported on Linux'
+    subdir_done()
+endif
+
+if (arch_subdir != 'x86' and arch_subdir != 'arm') or (not dpdk_conf.get('RTE_ARCH_64'))
+    build = false
+    reason = 'only supported on x86_64 and aarch64'
+    subdir_done()
+endif
+
+deps += ['bus_pci', 'dmadev']
+sources = files(
+        'hisi_dmadev.c',
+)
diff --git a/drivers/dma/hisilicon/version.map b/drivers/dma/hisilicon/version.map
new file mode 100644
index 0000000000..c2e0723b4c
--- /dev/null
+++ b/drivers/dma/hisilicon/version.map
@@ -0,0 +1,3 @@
+DPDK_22 {
+	local: *;
+};
diff --git a/drivers/dma/meson.build b/drivers/dma/meson.build
index a69418ce9b..6d6c054f51 100644
--- a/drivers/dma/meson.build
+++ b/drivers/dma/meson.build
@@ -4,6 +4,7 @@
 drivers = [
         'idxd',
         'ioat',
+        'hisilicon',
         'skeleton',
 ]
 std_deps = ['dmadev']
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
@ 2021-11-07 17:28     ` Thomas Monjalon
  0 siblings, 0 replies; 16+ messages in thread
From: Thomas Monjalon @ 2021-11-07 17:28 UTC (permalink / raw)
  To: Chengwen Feng; +Cc: dev

02/11/2021 13:37, Chengwen Feng:
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> +Hisilicon DMA
> +M: Chengwen Feng <fengchengwen@huawei.com>
> +F: drivers/dma/hisilicon

Final slash is missing (and it is mandatory for the parsing).

> --- a/doc/guides/dmadevs/index.rst
> +++ b/doc/guides/dmadevs/index.rst
> @@ -13,3 +13,4 @@ an application through DMA API.
>  
>     idxd
>     ioat
> +   hisilicon

Let's keep an alphabetical order.

> --- a/doc/guides/rel_notes/release_21_11.rst
> +++ b/doc/guides/rel_notes/release_21_11.rst
> +* **Added hisilicon dmadev driver implementation.**

HiSilicon DMA driver
blank line missing

> +  The hisilicon dmadev driver provide device drivers for the Kunpeng's DMA devices.
> +  This device driver can be used through the generic dmadev API.

> --- a/drivers/dma/meson.build
> +++ b/drivers/dma/meson.build
> @@ -4,6 +4,7 @@
>  drivers = [
>          'idxd',
>          'ioat',
> +        'hisilicon',
>          'skeleton',
>  ]

alphabetical order



^ permalink raw reply	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions Chengwen Feng
                     ` (4 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add dmadev instances create during the PCI probe, and
destroy them during the PCI remove. Internal structures and HW
definitions was also included.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 ++
 drivers/dma/hisilicon/hisi_dmadev.c | 212 +++++++++++++++++++++++++++-
 drivers/dma/hisilicon/hisi_dmadev.h |  97 +++++++++++++
 3 files changed, 318 insertions(+), 1 deletion(-)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 4cbaac4204..65138a8365 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -19,3 +19,13 @@ Device Setup
 
 Kunpeng DMA devices will need to be bound to a suitable DPDK-supported
 user-space IO driver such as ``vfio-pci`` in order to be used by DPDK.
+
+Device Probing and Initialization
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Once probed successfully, the device will appear as four ``dmadev`` which can be
+accessed using API from the ``rte_dmadev`` library.
+
+The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
+will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
+2nd ``dmadev`` name is "7b:00.0-ch1".
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index e6fb8a0fc8..b8369e7e71 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -6,7 +6,9 @@
 #include <string.h>
 
 #include <rte_bus_pci.h>
+#include <rte_cycles.h>
 #include <rte_eal.h>
+#include <rte_io.h>
 #include <rte_log.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
@@ -30,6 +32,141 @@ RTE_LOG_REGISTER_DEFAULT(hisi_dma_logtype, INFO);
 #define HISI_DMA_ERR(hw, fmt, args...) \
 		HISI_DMA_LOG_RAW(hw, ERR, fmt, ## args)
 
+static uint32_t
+hisi_dma_queue_base(struct hisi_dma_dev *hw)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08)
+		return HISI_DMA_HIP08_QUEUE_BASE;
+	else
+		return 0;
+}
+
+static void
+hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
+{
+	rte_write32(rte_cpu_to_le_32(val),
+		    (volatile void *)((char *)base + off));
+}
+
+static void
+hisi_dma_write_dev(struct hisi_dma_dev *hw, uint32_t off, uint32_t val)
+{
+	hisi_dma_write_reg(hw->io_base, off, val);
+}
+
+static void
+hisi_dma_write_queue(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t val)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	hisi_dma_write_dev(hw, off, val);
+}
+
+static uint32_t
+hisi_dma_read_reg(void *base, uint32_t off)
+{
+	uint32_t val = rte_read32((volatile void *)((char *)base + off));
+	return rte_le_to_cpu_32(val);
+}
+
+static uint32_t
+hisi_dma_read_dev(struct hisi_dma_dev *hw, uint32_t off)
+{
+	return hisi_dma_read_reg(hw->io_base, off);
+}
+
+static uint32_t
+hisi_dma_read_queue(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return hisi_dma_read_dev(hw, off);
+}
+
+static void
+hisi_dma_update_bit(struct hisi_dma_dev *hw, uint32_t off, uint32_t pos,
+		    bool set)
+{
+	uint32_t tmp = hisi_dma_read_dev(hw, off);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_dev(hw, off, tmp);
+}
+
+static void
+hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
+			  bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	uint32_t mask = 1u << pos;
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
+#define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
+	uint32_t timeout = 0; \
+	while (timeout++ <= (timeout_us)) { \
+		(val) = hisi_dma_read_queue(hw, HISI_DMA_QUEUE_FSM_REG); \
+		if (cond) \
+			break; \
+		rte_delay_us(sleep_us); \
+	} \
+	(cond) ? 0 : -ETIME; \
+})
+
+static int
+hisi_dma_reset_hw(struct hisi_dma_dev *hw)
+{
+#define POLL_SLEEP_US	100
+#define POLL_TIMEOUT_US	10000
+
+	uint32_t tmp;
+	int ret;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, true);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) != HISI_DMA_STATE_RUN,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "disable dma timeout!");
+		return ret;
+	}
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL1_REG,
+				  HISI_DMA_QUEUE_CTRL1_RESET_B, true);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_PAUSE_B, false);
+
+	ret = hisi_dma_poll_hw_state(hw, tmp,
+		FIELD_GET(HISI_DMA_QUEUE_FSM_STS_M, tmp) == HISI_DMA_STATE_IDLE,
+		POLL_SLEEP_US, POLL_TIMEOUT_US);
+	if (ret) {
+		HISI_DMA_ERR(hw, "reset dma timeout!");
+		return ret;
+	}
+
+	return 0;
+}
+
+static void
+hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
+{
+	struct hisi_dma_dev hw;
+
+	memset(&hw, 0, sizeof(hw));
+	hw.io_base = pci_bar;
+
+	if (revision == HISI_DMA_REVISION_HIP08B)
+		hisi_dma_update_bit(&hw, HISI_DMA_HIP08_MODE_REG,
+				    HISI_DMA_HIP08_MODE_SEL_B, true);
+}
+
 static uint8_t
 hisi_dma_reg_layout(uint8_t revision)
 {
@@ -49,6 +186,57 @@ hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function);
 }
 
+static void
+hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
+		      uint8_t queue_id, char *name, size_t size)
+{
+	memset(name, 0, size);
+	(void)snprintf(name, size, "%x:%x.%x-ch%u",
+		 pci_dev->addr.bus, pci_dev->addr.devid,
+		 pci_dev->addr.function, queue_id);
+}
+
+static int
+hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
+		uint8_t revision)
+{
+#define REG_PCI_BAR_INDEX	2
+
+	char name[RTE_DEV_NAME_MAX_LEN];
+	struct rte_dma_dev *dev;
+	struct hisi_dma_dev *hw;
+	int ret;
+
+	hisi_dma_gen_dev_name(pci_dev, queue_id, name, sizeof(name));
+	dev = rte_dma_pmd_allocate(name, pci_dev->device.numa_node,
+				   sizeof(*hw));
+	if (dev == NULL) {
+		HISI_DMA_LOG(ERR, "%s allocate dmadev fail!", name);
+		return -EINVAL;
+	}
+
+	dev->device = &pci_dev->device;
+
+	hw = dev->data->dev_private;
+	hw->data = dev->data;
+	hw->revision = revision;
+	hw->reg_layout = hisi_dma_reg_layout(revision);
+	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
+	hw->queue_id = queue_id;
+
+	ret = hisi_dma_reset_hw(hw);
+	if (ret) {
+		HISI_DMA_LOG(ERR, "%s init device fail!", name);
+		(void)rte_dma_pmd_release(name);
+		return -EIO;
+	}
+
+	dev->state = RTE_DMA_DEV_READY;
+	HISI_DMA_LOG(DEBUG, "%s create dmadev success!", name);
+
+	return 0;
+}
+
 static int
 hisi_dma_check_revision(struct rte_pci_device *pci_dev, const char *name,
 			uint8_t *out_revision)
@@ -78,6 +266,7 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 {
 	char name[RTE_DEV_NAME_MAX_LEN] = { 0 };
 	uint8_t revision;
+	uint8_t i;
 	int ret;
 
 	hisi_dma_gen_pci_device_name(pci_dev, name, sizeof(name));
@@ -92,13 +281,34 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
+	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		ret = hisi_dma_create(pci_dev, i, revision);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s create dmadev %u failed!",
+				     name, i);
+			break;
+		}
+	}
+
 	return ret;
 }
 
 static int
 hisi_dma_remove(struct rte_pci_device *pci_dev)
 {
-	RTE_SET_USED(pci_dev);
+	char name[RTE_DEV_NAME_MAX_LEN];
+	uint8_t i;
+	int ret;
+
+	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
+		hisi_dma_gen_dev_name(pci_dev, i, name, sizeof(name));
+		ret = rte_dma_pmd_release(name);
+		if (ret)
+			return ret;
+	}
+
 	return 0;
 }
 
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 114b9dcb5b..50aaa38b72 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -5,11 +5,24 @@
 #ifndef HISI_DMADEV_H
 #define HISI_DMADEV_H
 
+#include <rte_byteorder.h>
+#include <rte_common.h>
+
+#define BIT(x)	(1ul << (x))
+#define BITS_PER_LONG	(__SIZEOF_LONG__ * 8)
+#define GENMASK(h, l) \
+		(((~0UL) << (l)) & (~0UL >> (BITS_PER_LONG - 1 - (h))))
+#define BF_SHF(x) (__builtin_ffsll(x) - 1)
+#define FIELD_GET(mask, reg) \
+		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
+#define HISI_DMA_MAX_HW_QUEUES			4
+
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
  * they have the same pci device id but with different pci revision.
@@ -21,4 +34,88 @@ enum {
 	HISI_DMA_REG_LAYOUT_HIP08
 };
 
+/**
+ * Hardware PCI bar register MAP:
+ *
+ *     --------------
+ *     | Misc-reg-0 |
+ *     |            |
+ *     --------------   -> Queue base
+ *     |            |
+ *     | Queue-0    |
+ *     |            |
+ *     --------------   ---
+ *     |            |    ^
+ *     | Queue-1    |   Queue region
+ *     |            |    v
+ *     --------------   ---
+ *     | ...        |
+ *     | Queue-x    |
+ *     | ...        |
+ *     --------------
+ *     | Misc-reg-1 |
+ *     --------------
+ *
+ * As described above, a single queue register is continuous and occupies the
+ * length of queue-region. The global offset for a single queue register is
+ * calculated by:
+ *     offset = queue-base + (queue-id * queue-region) + reg-offset-in-region.
+ *
+ * The first part of queue region is basically the same for HIP08 and later chip
+ * register layouts, therefore, HISI_QUEUE_* registers are defined for it.
+ */
+#define HISI_DMA_QUEUE_SQ_BASE_L_REG		0x0
+#define HISI_DMA_QUEUE_SQ_BASE_H_REG		0x4
+#define HISI_DMA_QUEUE_SQ_DEPTH_REG		0x8
+#define HISI_DMA_QUEUE_SQ_TAIL_REG		0xC
+#define HISI_DMA_QUEUE_CQ_BASE_L_REG		0x10
+#define HISI_DMA_QUEUE_CQ_BASE_H_REG		0x14
+#define HISI_DMA_QUEUE_CQ_DEPTH_REG		0x18
+#define HISI_DMA_QUEUE_CQ_HEAD_REG		0x1C
+#define HISI_DMA_QUEUE_CTRL0_REG		0x20
+#define HISI_DMA_QUEUE_CTRL0_EN_B		0
+#define HISI_DMA_QUEUE_CTRL0_PAUSE_B		4
+#define HISI_DMA_QUEUE_CTRL1_REG		0x24
+#define HISI_DMA_QUEUE_CTRL1_RESET_B		0
+#define HISI_DMA_QUEUE_FSM_REG			0x30
+#define HISI_DMA_QUEUE_FSM_STS_M		GENMASK(3, 0)
+#define HISI_DMA_QUEUE_INT_STATUS_REG		0x40
+#define HISI_DMA_QUEUE_ERR_INT_NUM0_REG		0x84
+#define HISI_DMA_QUEUE_ERR_INT_NUM1_REG		0x88
+#define HISI_DMA_QUEUE_ERR_INT_NUM2_REG		0x8C
+#define HISI_DMA_QUEUE_REGION_SIZE		0x100
+
+/**
+ * HiSilicon IP08 DMA register and field define:
+ */
+#define HISI_DMA_HIP08_QUEUE_BASE			0x0
+#define HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B		2
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_REG		0x44
+#define HISI_DMA_HIP08_QUEUE_INT_MASK_M			GENMASK(14, 0)
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG		0x90
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG		0x94
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG		0x98
+#define HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG		0x48
+#define HISI_DMA_HIP08_MODE_REG				0x217C
+#define HISI_DMA_HIP08_MODE_SEL_B			0
+#define HISI_DMA_HIP08_DUMP_START_REG			0x2000
+#define HISI_DMA_HIP08_DUMP_END_REG			0x2280
+
+/**
+ * In fact, there are multiple states, but it need to pay attention to
+ * the following two states for the driver:
+ */
+enum {
+	HISI_DMA_STATE_IDLE = 0,
+	HISI_DMA_STATE_RUN,
+};
+
+struct hisi_dma_dev {
+	struct rte_dma_dev_data *data;
+	uint8_t revision; /**< PCI revision. */
+	uint8_t reg_layout; /**< hardware register layout. */
+	void *io_base;
+	uint8_t queue_id; /**< hardware DMA queue index. */
+};
+
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data " Chengwen Feng
                     ` (3 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add control path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 doc/guides/dmadevs/hisilicon.rst    |  10 +
 drivers/dma/hisilicon/hisi_dmadev.c | 385 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  99 +++++++
 3 files changed, 494 insertions(+)

diff --git a/doc/guides/dmadevs/hisilicon.rst b/doc/guides/dmadevs/hisilicon.rst
index 65138a8365..24bae86bdc 100644
--- a/doc/guides/dmadevs/hisilicon.rst
+++ b/doc/guides/dmadevs/hisilicon.rst
@@ -29,3 +29,13 @@ accessed using API from the ``rte_dmadev`` library.
 The name of the ``dmadev`` created is like "B:D.F-chX", e.g. DMA 0000:7b:00.0
 will create four ``dmadev``, the 1st ``dmadev`` name is "7b:00.0-ch0", and the
 2nd ``dmadev`` name is "7b:00.0-ch1".
+
+Device Configuration
+~~~~~~~~~~~~~~~~~~~~~
+
+Kunpeng DMA configuration requirements:
+
+* ``ring_size`` must be a power of two, between 32 and 8192.
+* Only one ``vchan`` is supported per ``dmadev``.
+* Silent mode is not supported.
+* The transfer direction must be set to ``RTE_DMA_DIR_MEM_TO_MEM``.
diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index b8369e7e71..bcdcf4de4b 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -10,6 +10,8 @@
 #include <rte_eal.h>
 #include <rte_io.h>
 #include <rte_log.h>
+#include <rte_malloc.h>
+#include <rte_memzone.h>
 #include <rte_pci.h>
 #include <rte_dmadev_pmd.h>
 
@@ -41,6 +43,14 @@ hisi_dma_queue_base(struct hisi_dma_dev *hw)
 		return 0;
 }
 
+static volatile void *
+hisi_dma_queue_regaddr(struct hisi_dma_dev *hw, uint32_t qoff)
+{
+	uint32_t off = hisi_dma_queue_base(hw) +
+			hw->queue_id * HISI_DMA_QUEUE_REGION_SIZE + qoff;
+	return (volatile void *)((char *)hw->io_base + off);
+}
+
 static void
 hisi_dma_write_reg(void *base, uint32_t off, uint32_t val)
 {
@@ -103,6 +113,15 @@ hisi_dma_update_queue_bit(struct hisi_dma_dev *hw, uint32_t qoff, uint32_t pos,
 	hisi_dma_write_queue(hw, qoff, tmp);
 }
 
+static void
+hisi_dma_update_queue_mbit(struct hisi_dma_dev *hw, uint32_t qoff,
+			   uint32_t mask, bool set)
+{
+	uint32_t tmp = hisi_dma_read_queue(hw, qoff);
+	tmp = set ? tmp | mask : tmp & ~mask;
+	hisi_dma_write_queue(hw, qoff, tmp);
+}
+
 #define hisi_dma_poll_hw_state(hw, val, cond, sleep_us, timeout_us) ({ \
 	uint32_t timeout = 0; \
 	while (timeout++ <= (timeout_us)) { \
@@ -154,6 +173,45 @@ hisi_dma_reset_hw(struct hisi_dma_dev *hw)
 	return 0;
 }
 
+static void
+hisi_dma_init_hw(struct hisi_dma_dev *hw)
+{
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_L_REG,
+			     lower_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_BASE_H_REG,
+			     upper_32_bits(hw->sqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_L_REG,
+			     lower_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_BASE_H_REG,
+			     upper_32_bits(hw->cqe_iova));
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_DEPTH_REG,
+			     hw->sq_depth_mask);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_DEPTH_REG, hw->cq_depth - 1);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_SQ_TAIL_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_CQ_HEAD_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM0_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM1_REG, 0);
+	hisi_dma_write_queue(hw, HISI_DMA_QUEUE_ERR_INT_NUM2_REG, 0);
+
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM3_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM4_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM5_REG,
+				     0);
+		hisi_dma_write_queue(hw, HISI_DMA_HIP08_QUEUE_ERR_INT_NUM6_REG,
+				     0);
+		hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				HISI_DMA_HIP08_QUEUE_CTRL0_ERR_ABORT_B, false);
+		hisi_dma_update_queue_mbit(hw, HISI_DMA_QUEUE_INT_STATUS_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+		hisi_dma_update_queue_mbit(hw,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_REG,
+				HISI_DMA_HIP08_QUEUE_INT_MASK_M, true);
+	}
+}
+
 static void
 hisi_dma_init_gbl(void *pci_bar, uint8_t revision)
 {
@@ -176,6 +234,301 @@ hisi_dma_reg_layout(uint8_t revision)
 		return HISI_DMA_REG_LAYOUT_INVALID;
 }
 
+static void
+hisi_dma_zero_iomem(struct hisi_dma_dev *hw)
+{
+	memset(hw->iomz->addr, 0, hw->iomz_sz);
+}
+
+static int
+hisi_dma_alloc_iomem(struct hisi_dma_dev *hw, uint16_t ring_size,
+		     const char *dev_name)
+{
+	uint32_t sq_size = sizeof(struct hisi_dma_sqe) * ring_size;
+	uint32_t cq_size = sizeof(struct hisi_dma_cqe) *
+			   (ring_size + HISI_DMA_CQ_RESERVED);
+	uint32_t status_size = sizeof(uint16_t) * ring_size;
+	char mz_name[RTE_MEMZONE_NAMESIZE];
+	const struct rte_memzone *iomz;
+	uint32_t total_size;
+
+	sq_size = RTE_CACHE_LINE_ROUNDUP(sq_size);
+	cq_size = RTE_CACHE_LINE_ROUNDUP(cq_size);
+	status_size = RTE_CACHE_LINE_ROUNDUP(status_size);
+	total_size = sq_size + cq_size + status_size;
+
+	(void)snprintf(mz_name, sizeof(mz_name), "hisi_dma:%s", dev_name);
+	iomz = rte_memzone_reserve(mz_name, total_size, hw->data->numa_node,
+				   RTE_MEMZONE_IOVA_CONTIG);
+	if (iomz == NULL) {
+		HISI_DMA_ERR(hw, "malloc %s iomem fail!", mz_name);
+		return -ENOMEM;
+	}
+
+	hw->iomz = iomz;
+	hw->iomz_sz = total_size;
+	hw->sqe = iomz->addr;
+	hw->cqe = (void *)((char *)iomz->addr + sq_size);
+	hw->status = (void *)((char *)iomz->addr + sq_size + cq_size);
+	hw->sqe_iova = iomz->iova;
+	hw->cqe_iova = iomz->iova + sq_size;
+	hw->sq_depth_mask = ring_size - 1;
+	hw->cq_depth = ring_size + HISI_DMA_CQ_RESERVED;
+	hisi_dma_zero_iomem(hw);
+
+	return 0;
+}
+
+static void
+hisi_dma_free_iomem(struct hisi_dma_dev *hw)
+{
+	if (hw->iomz != NULL)
+		rte_memzone_free(hw->iomz);
+
+	hw->iomz = NULL;
+	hw->sqe = NULL;
+	hw->cqe = NULL;
+	hw->status = NULL;
+	hw->sqe_iova = 0;
+	hw->cqe_iova = 0;
+	hw->sq_depth_mask = 0;
+	hw->cq_depth = 0;
+}
+
+static int
+hisi_dma_info_get(const struct rte_dma_dev *dev,
+		  struct rte_dma_info *dev_info,
+		  uint32_t info_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(info_sz);
+
+	dev_info->dev_capa = RTE_DMA_CAPA_MEM_TO_MEM |
+			     RTE_DMA_CAPA_OPS_COPY;
+	dev_info->max_vchans = 1;
+	dev_info->max_desc = HISI_DMA_MAX_DESC_NUM;
+	dev_info->min_desc = HISI_DMA_MIN_DESC_NUM;
+
+	return 0;
+}
+
+static int
+hisi_dma_configure(struct rte_dma_dev *dev,
+		   const struct rte_dma_conf *conf,
+		   uint32_t conf_sz)
+{
+	RTE_SET_USED(dev);
+	RTE_SET_USED(conf);
+	RTE_SET_USED(conf_sz);
+	return 0;
+}
+
+static int
+hisi_dma_vchan_setup(struct rte_dma_dev *dev, uint16_t vchan,
+		     const struct rte_dma_vchan_conf *conf,
+		     uint32_t conf_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+	int ret;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(conf_sz);
+
+	if (!rte_is_power_of_2(conf->nb_desc)) {
+		HISI_DMA_ERR(hw, "Number of desc must be power of 2!");
+		return -EINVAL;
+	}
+
+	hisi_dma_free_iomem(hw);
+	ret = hisi_dma_alloc_iomem(hw, conf->nb_desc, dev->data->dev_name);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static int
+hisi_dma_start(struct rte_dma_dev *dev)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	if (hw->iomz == NULL) {
+		HISI_DMA_ERR(hw, "Vchan was not setup, start fail!\n");
+		return -EINVAL;
+	}
+
+	/* Reset the dmadev to a known state, include:
+	 *   1) zero iomem, also include status fields.
+	 *   2) init hardware register.
+	 *   3) init index values to zero.
+	 *   4) init running statistics.
+	 */
+	hisi_dma_zero_iomem(hw);
+	hisi_dma_init_hw(hw);
+	hw->ridx = 0;
+	hw->cridx = 0;
+	hw->sq_head = 0;
+	hw->sq_tail = 0;
+	hw->cq_sq_head = 0;
+	hw->cq_head = 0;
+	hw->cqs_completed = 0;
+	hw->cqe_vld = 1;
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	hisi_dma_update_queue_bit(hw, HISI_DMA_QUEUE_CTRL0_REG,
+				  HISI_DMA_QUEUE_CTRL0_EN_B, true);
+
+	return 0;
+}
+
+static int
+hisi_dma_stop(struct rte_dma_dev *dev)
+{
+	return hisi_dma_reset_hw(dev->data->dev_private);
+}
+
+static int
+hisi_dma_close(struct rte_dma_dev *dev)
+{
+	/* The dmadev already stopped */
+	hisi_dma_free_iomem(dev->data->dev_private);
+	return 0;
+}
+
+static int
+hisi_dma_stats_get(const struct rte_dma_dev *dev, uint16_t vchan,
+		   struct rte_dma_stats *stats,
+		   uint32_t stats_sz)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	RTE_SET_USED(stats_sz);
+	stats->submitted = hw->submitted;
+	stats->completed = hw->completed;
+	stats->errors = hw->errors;
+
+	return 0;
+}
+
+static int
+hisi_dma_stats_reset(struct rte_dma_dev *dev, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	RTE_SET_USED(vchan);
+	hw->submitted = 0;
+	hw->completed = 0;
+	hw->errors = 0;
+
+	return 0;
+}
+
+static void
+hisi_dma_get_dump_range(struct hisi_dma_dev *hw, uint32_t *start, uint32_t *end)
+{
+	if (hw->reg_layout == HISI_DMA_REG_LAYOUT_HIP08) {
+		*start = HISI_DMA_HIP08_DUMP_START_REG;
+		*end = HISI_DMA_HIP08_DUMP_END_REG;
+	} else {
+		*start = 0;
+		*end = 0;
+	}
+}
+
+static void
+hisi_dma_dump_common(struct hisi_dma_dev *hw, FILE *f)
+{
+#define DUMP_REGNUM_PER_LINE	4
+
+	uint32_t start, end;
+	uint32_t cnt, i;
+
+	hisi_dma_get_dump_range(hw, &start, &end);
+
+	(void)fprintf(f, "    common-register:\n");
+
+	cnt = 0;
+	for (i = start; i <= end; i += sizeof(uint32_t)) {
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "      [%4x]:", i);
+		(void)fprintf(f, " 0x%08x", hisi_dma_read_dev(hw, i));
+		cnt++;
+		if (cnt % DUMP_REGNUM_PER_LINE == 0)
+			(void)fprintf(f, "\n");
+	}
+	if (cnt % DUMP_REGNUM_PER_LINE)
+		(void)fprintf(f, "\n");
+}
+
+static void
+hisi_dma_dump_read_queue(struct hisi_dma_dev *hw, uint32_t qoff,
+			 char *buffer, int max_sz)
+{
+	memset(buffer, 0, max_sz);
+
+	/* Address-related registers are not printed for security reasons. */
+	if (qoff == HISI_DMA_QUEUE_SQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_SQ_BASE_H_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_L_REG ||
+	    qoff == HISI_DMA_QUEUE_CQ_BASE_H_REG) {
+		(void)snprintf(buffer, max_sz, "**********");
+		return;
+	}
+
+	(void)snprintf(buffer, max_sz, "0x%08x", hisi_dma_read_queue(hw, qoff));
+}
+
+static void
+hisi_dma_dump_queue(struct hisi_dma_dev *hw, FILE *f)
+{
+#define REG_FMT_LEN	32
+	char buf[REG_FMT_LEN] = { 0 };
+	uint32_t i;
+
+	(void)fprintf(f, "    queue-register:\n");
+	for (i = 0; i < HISI_DMA_QUEUE_REGION_SIZE; ) {
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, "      [%2x]: %s", i, buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s", buf);
+		i += sizeof(uint32_t);
+		hisi_dma_dump_read_queue(hw, i, buf, sizeof(buf));
+		(void)fprintf(f, " %s\n", buf);
+		i += sizeof(uint32_t);
+	}
+}
+
+static int
+hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
+{
+	struct hisi_dma_dev *hw = dev->data->dev_private;
+
+	(void)fprintf(f,
+		"    revision: 0x%x queue_id: %u ring_size: %u\n"
+		"    ridx: %u cridx: %u\n"
+		"    sq_head: %u sq_tail: %u cq_sq_head: %u\n"
+		"    cq_head: %u cqs_completed: %u cqe_vld: %u\n"
+		"    submitted: %" PRIu64 " completed: %" PRIu64 " errors %"
+		PRIu64"\n",
+		hw->revision, hw->queue_id,
+		hw->sq_depth_mask > 0 ? hw->sq_depth_mask + 1 : 0,
+		hw->ridx, hw->cridx,
+		hw->sq_head, hw->sq_tail, hw->cq_sq_head,
+		hw->cq_head, hw->cqs_completed, hw->cqe_vld,
+		hw->submitted, hw->completed, hw->errors);
+	hisi_dma_dump_queue(hw, f);
+	hisi_dma_dump_common(hw, f);
+
+	return 0;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -196,6 +549,33 @@ hisi_dma_gen_dev_name(const struct rte_pci_device *pci_dev,
 		 pci_dev->addr.function, queue_id);
 }
 
+/**
+ * Hardware queue state machine:
+ *
+ *   -----------  dmadev_create	  ------------------
+ *   | Unknown | ---------------> |      IDLE      |
+ *   -----------                  ------------------
+ *                                   ^          |
+ *                                   |          |dev_start
+ *                           dev_stop|          |
+ *                                   |          v
+ *                                ------------------
+ *                                |      RUN       |
+ *                                ------------------
+ *
+ */
+static const struct rte_dma_dev_ops hisi_dmadev_ops = {
+	.dev_info_get     = hisi_dma_info_get,
+	.dev_configure    = hisi_dma_configure,
+	.dev_start        = hisi_dma_start,
+	.dev_stop         = hisi_dma_stop,
+	.dev_close        = hisi_dma_close,
+	.vchan_setup      = hisi_dma_vchan_setup,
+	.stats_get        = hisi_dma_stats_get,
+	.stats_reset      = hisi_dma_stats_reset,
+	.dev_dump         = hisi_dma_dump,
+};
+
 static int
 hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 		uint8_t revision)
@@ -216,6 +596,7 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	}
 
 	dev->device = &pci_dev->device;
+	dev->dev_ops = &hisi_dmadev_ops;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
@@ -223,6 +604,10 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->reg_layout = hisi_dma_reg_layout(revision);
 	hw->io_base = pci_dev->mem_resource[REG_PCI_BAR_INDEX].addr;
 	hw->queue_id = queue_id;
+	hw->sq_tail_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_SQ_TAIL_REG);
+	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
+						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
 	ret = hisi_dma_reset_hw(hw);
 	if (ret) {
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index 50aaa38b72..dd0315cd31 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -16,12 +16,17 @@
 #define FIELD_GET(mask, reg) \
 		((typeof(mask))(((reg) & (mask)) >> BF_SHF(mask)))
 
+#define lower_32_bits(x) ((uint32_t)(x))
+#define upper_32_bits(x) ((uint32_t)(((x) >> 16) >> 16))
+
 #define PCI_VENDOR_ID_HUAWEI			0x19e5
 #define HISI_DMA_DEVICE_ID			0xA122
 #define HISI_DMA_PCI_REVISION_ID_REG		0x08
 #define HISI_DMA_REVISION_HIP08B		0x21
 
 #define HISI_DMA_MAX_HW_QUEUES			4
+#define HISI_DMA_MAX_DESC_NUM			8192
+#define HISI_DMA_MIN_DESC_NUM			32
 
 /**
  * The HIP08B(HiSilicon IP08) and later Chip(e.g. HiSilicon IP09) are DMA iEPs,
@@ -110,12 +115,106 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * After scanning the CQ array, the CQ head register needs to be updated.
+ * Updating the register involves write memory barrier operations.
+ * Here use the following method to reduce WMB operations:
+ *   a) malloc more CQEs, which correspond to the macro HISI_DMA_CQ_RESERVED.
+ *   b) update the CQ head register after accumulated number of completed CQs
+ *      is greater than or equal to HISI_DMA_CQ_RESERVED.
+ */
+#define HISI_DMA_CQ_RESERVED		64
+
+struct hisi_dma_sqe {
+	uint32_t dw0;
+#define SQE_FENCE_FLAG	BIT(10)
+#define SQE_OPCODE_M2M	0x4
+	uint32_t dw1;
+	uint32_t dw2;
+	uint32_t length;
+	uint64_t src_addr;
+	uint64_t dst_addr;
+};
+
+struct hisi_dma_cqe {
+	uint64_t rsv;
+	uint64_t misc;
+#define CQE_SQ_HEAD_MASK	GENMASK(15, 0)
+#define CQE_VALID_B		BIT(48)
+#define CQE_STATUS_MASK		GENMASK(63, 49)
+};
+
 struct hisi_dma_dev {
+	struct hisi_dma_sqe *sqe;
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t *status; /* the completion status array of SQEs. */
+
+	volatile void *sq_tail_reg; /**< register address for doorbell. */
+	volatile void *cq_head_reg; /**< register address for answer CQ. */
+
+	uint16_t sq_depth_mask; /**< SQ depth - 1, the SQ depth is power of 2 */
+	uint16_t cq_depth; /* CQ depth */
+
+	uint16_t ridx; /**< ring index which will assign to the next request. */
+	/** ring index which returned by hisi_dmadev_completed APIs. */
+	uint16_t cridx;
+
+	/**
+	 * SQE array management fields:
+	 *
+	 *  -----------------------------------------------------
+	 *  | SQE0 | SQE1 | SQE2 |   ...  | SQEx | ... | SQEn-1 |
+	 *  -----------------------------------------------------
+	 *     ^             ^               ^
+	 *     |             |               |
+	 *   sq_head     cq_sq_head       sq_tail
+	 *
+	 *  sq_head: index to the oldest completed request, this filed was
+	 *           updated by hisi_dmadev_completed* APIs.
+	 *  sq_tail: index of the next new request, this field was updated by
+	 *           hisi_dmadev_copy API.
+	 *  cq_sq_head: next index of index that has been completed by hardware,
+	 *              this filed was updated by hisi_dmadev_completed* APIs.
+	 *
+	 *  [sq_head, cq_sq_head): the SQEs that hardware already completed.
+	 *  [cq_sq_head, sq_tail): the SQEs that hardware processing.
+	 */
+	uint16_t sq_head;
+	uint16_t sq_tail;
+	uint16_t cq_sq_head;
+	/**
+	 * The driver scans the CQE array, if the valid bit changes, the CQE is
+	 * considered valid.
+	 * Note: One CQE is corresponding to one or several SQEs, e.g. app
+	 *       submits two copy requests, the hardware processes the two SQEs,
+	 *       but it may write back only one CQE and the CQE's sq_head field
+	 *       indicates the index of the second copy request in the SQE
+	 *       array.
+	 */
+	uint16_t cq_head; /**< CQ index for next scans. */
+	/** accumulated number of completed CQs
+	 * @see HISI_DMA_CQ_RESERVED
+	 */
+	uint16_t cqs_completed;
+	uint8_t cqe_vld; /**< valid bit for CQE, will change for every round. */
+
+	uint64_t submitted;
+	uint64_t completed;
+	uint64_t errors;
+
+	/**
+	 * The following fields are not accessed in the I/O path, so they are
+	 * placed at the end.
+	 */
 	struct rte_dma_dev_data *data;
 	uint8_t revision; /**< PCI revision. */
 	uint8_t reg_layout; /**< hardware register layout. */
 	void *io_base;
 	uint8_t queue_id; /**< hardware DMA queue index. */
+	const struct rte_memzone *iomz;
+	uint32_t iomz_sz;
+	rte_iova_t sqe_iova;
+	rte_iova_t cqe_iova;
 };
 
 #endif /* HISI_DMADEV_H */
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data path functions
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (2 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process Chengwen Feng
                     ` (2 subsequent siblings)
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add data path functions for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 206 ++++++++++++++++++++++++++++
 drivers/dma/hisilicon/hisi_dmadev.h |  16 +++
 2 files changed, 222 insertions(+)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index bcdcf4de4b..d03967cae3 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -529,6 +529,206 @@ hisi_dma_dump(const struct rte_dma_dev *dev, FILE *f)
 	return 0;
 }
 
+static int
+hisi_dma_copy(void *dev_private, uint16_t vchan,
+		 rte_iova_t src, rte_iova_t dst,
+		 uint32_t length, uint64_t flags)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	struct hisi_dma_sqe *sqe = &hw->sqe[hw->sq_tail];
+
+	RTE_SET_USED(vchan);
+
+	if (((hw->sq_tail + 1) & hw->sq_depth_mask) == hw->sq_head)
+		return -ENOSPC;
+
+	sqe->dw0 = rte_cpu_to_le_32(SQE_OPCODE_M2M);
+	sqe->dw1 = 0;
+	sqe->dw2 = 0;
+	sqe->length = rte_cpu_to_le_32(length);
+	sqe->src_addr = rte_cpu_to_le_64(src);
+	sqe->dst_addr = rte_cpu_to_le_64(dst);
+	hw->sq_tail = (hw->sq_tail + 1) & hw->sq_depth_mask;
+	hw->submitted++;
+
+	if (flags & RTE_DMA_OP_FLAG_FENCE)
+		sqe->dw0 |= rte_cpu_to_le_32(SQE_FENCE_FLAG);
+	if (flags & RTE_DMA_OP_FLAG_SUBMIT)
+		rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return hw->ridx++;
+}
+
+static int
+hisi_dma_submit(void *dev_private, uint16_t vchan)
+{
+	struct hisi_dma_dev *hw = dev_private;
+
+	RTE_SET_USED(vchan);
+	rte_write32(rte_cpu_to_le_32(hw->sq_tail), hw->sq_tail_reg);
+
+	return 0;
+}
+
+static inline void
+hisi_dma_scan_cq(struct hisi_dma_dev *hw)
+{
+	volatile struct hisi_dma_cqe *cqe;
+	uint16_t csq_head = hw->cq_sq_head;
+	uint16_t cq_head = hw->cq_head;
+	uint16_t count = 0;
+	uint64_t misc;
+
+	while (true) {
+		cqe = &hw->cqe[cq_head];
+		misc = cqe->misc;
+		misc = rte_le_to_cpu_64(misc);
+		if (FIELD_GET(CQE_VALID_B, misc) != hw->cqe_vld)
+			break;
+
+		csq_head = FIELD_GET(CQE_SQ_HEAD_MASK, misc);
+		if (unlikely(misc & CQE_STATUS_MASK))
+			hw->status[csq_head] = FIELD_GET(CQE_STATUS_MASK,
+							 misc);
+
+		count++;
+		cq_head++;
+		if (cq_head == hw->cq_depth) {
+			hw->cqe_vld = !hw->cqe_vld;
+			cq_head = 0;
+		}
+	}
+
+	if (count == 0)
+		return;
+
+	hw->cq_head = cq_head;
+	hw->cq_sq_head = (csq_head + 1) & hw->sq_depth_mask;
+	hw->cqs_completed += count;
+	if (hw->cqs_completed >= HISI_DMA_CQ_RESERVED) {
+		rte_write32(rte_cpu_to_le_32(cq_head), hw->cq_head_reg);
+		hw->cqs_completed = 0;
+	}
+}
+
+static inline uint16_t
+hisi_dma_calc_cpls(struct hisi_dma_dev *hw, const uint16_t nb_cpls)
+{
+	uint16_t cpl_num;
+
+	if (hw->cq_sq_head >= hw->sq_head)
+		cpl_num = hw->cq_sq_head - hw->sq_head;
+	else
+		cpl_num = hw->sq_depth_mask + 1 - hw->sq_head + hw->cq_sq_head;
+
+	if (cpl_num > nb_cpls)
+		cpl_num = nb_cpls;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_completed(void *dev_private,
+		   uint16_t vchan, const uint16_t nb_cpls,
+		   uint16_t *last_idx, bool *has_error)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		if (hw->status[sq_head]) {
+			*has_error = true;
+			break;
+		}
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (i > 0) {
+		hw->cridx += i;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += i;
+
+	return i;
+}
+
+static enum rte_dma_status_code
+hisi_dma_convert_status(uint16_t status)
+{
+	switch (status) {
+	case HISI_DMA_STATUS_SUCCESS:
+		return RTE_DMA_STATUS_SUCCESSFUL;
+	case HISI_DMA_STATUS_INVALID_OPCODE:
+		return RTE_DMA_STATUS_INVALID_OPCODE;
+	case HISI_DMA_STATUS_INVALID_LENGTH:
+		return RTE_DMA_STATUS_INVALID_LENGTH;
+	case HISI_DMA_STATUS_USER_ABORT:
+		return RTE_DMA_STATUS_USER_ABORT;
+	case HISI_DMA_STATUS_REMOTE_READ_ERROR:
+	case HISI_DMA_STATUS_AXI_READ_ERROR:
+		return RTE_DMA_STATUS_BUS_READ_ERROR;
+	case HISI_DMA_STATUS_AXI_WRITE_ERROR:
+		return RTE_DMA_STATUS_BUS_WRITE_ERROR;
+	case HISI_DMA_STATUS_DATA_POISON:
+	case HISI_DMA_STATUS_REMOTE_DATA_POISION:
+		return RTE_DMA_STATUS_DATA_POISION;
+	case HISI_DMA_STATUS_SQE_READ_ERROR:
+	case HISI_DMA_STATUS_SQE_READ_POISION:
+		return RTE_DMA_STATUS_DESCRIPTOR_READ_ERROR;
+	case HISI_DMA_STATUS_LINK_DOWN_ERROR:
+		return RTE_DMA_STATUS_DEV_LINK_ERROR;
+	default:
+		return RTE_DMA_STATUS_ERROR_UNKNOWN;
+	}
+}
+
+static uint16_t
+hisi_dma_completed_status(void *dev_private,
+			  uint16_t vchan, const uint16_t nb_cpls,
+			  uint16_t *last_idx, enum rte_dma_status_code *status)
+{
+	struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t cpl_num, i;
+
+	RTE_SET_USED(vchan);
+	hisi_dma_scan_cq(hw);
+
+	cpl_num = hisi_dma_calc_cpls(hw, nb_cpls);
+	for (i = 0; i < cpl_num; i++) {
+		status[i] = hisi_dma_convert_status(hw->status[sq_head]);
+		hw->errors += !!status[i];
+		hw->status[sq_head] = HISI_DMA_STATUS_SUCCESS;
+		sq_head = (sq_head + 1) & hw->sq_depth_mask;
+	}
+	if (likely(cpl_num > 0)) {
+		hw->cridx += cpl_num;
+		*last_idx = hw->cridx - 1;
+		hw->sq_head = sq_head;
+	}
+	hw->completed += cpl_num;
+
+	return cpl_num;
+}
+
+static uint16_t
+hisi_dma_burst_capacity(const void *dev_private, uint16_t vchan)
+{
+	const struct hisi_dma_dev *hw = dev_private;
+	uint16_t sq_head = hw->sq_head;
+	uint16_t sq_tail = hw->sq_tail;
+
+	RTE_SET_USED(vchan);
+
+	return (sq_tail >= sq_head) ? hw->sq_depth_mask - sq_tail + sq_head :
+				      sq_head - 1 - sq_tail;
+}
+
 static void
 hisi_dma_gen_pci_device_name(const struct rte_pci_device *pci_dev,
 			     char *name, size_t size)
@@ -597,6 +797,12 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 
 	dev->device = &pci_dev->device;
 	dev->dev_ops = &hisi_dmadev_ops;
+	dev->fp_obj->dev_private = dev->data->dev_private;
+	dev->fp_obj->copy = hisi_dma_copy;
+	dev->fp_obj->submit = hisi_dma_submit;
+	dev->fp_obj->completed = hisi_dma_completed;
+	dev->fp_obj->completed_status = hisi_dma_completed_status;
+	dev->fp_obj->burst_capacity = hisi_dma_burst_capacity;
 
 	hw = dev->data->dev_private;
 	hw->data = dev->data;
diff --git a/drivers/dma/hisilicon/hisi_dmadev.h b/drivers/dma/hisilicon/hisi_dmadev.h
index dd0315cd31..12e209c86e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.h
+++ b/drivers/dma/hisilicon/hisi_dmadev.h
@@ -115,6 +115,22 @@ enum {
 	HISI_DMA_STATE_RUN,
 };
 
+/**
+ * Hardware complete status define:
+ */
+#define HISI_DMA_STATUS_SUCCESS			0x0
+#define HISI_DMA_STATUS_INVALID_OPCODE		0x1
+#define HISI_DMA_STATUS_INVALID_LENGTH		0x2
+#define HISI_DMA_STATUS_USER_ABORT		0x4
+#define HISI_DMA_STATUS_REMOTE_READ_ERROR	0x10
+#define HISI_DMA_STATUS_AXI_READ_ERROR		0x20
+#define HISI_DMA_STATUS_AXI_WRITE_ERROR		0x40
+#define HISI_DMA_STATUS_DATA_POISON		0x80
+#define HISI_DMA_STATUS_SQE_READ_ERROR		0x100
+#define HISI_DMA_STATUS_SQE_READ_POISION	0x200
+#define HISI_DMA_STATUS_REMOTE_DATA_POISION	0x400
+#define HISI_DMA_STATUS_LINK_DOWN_ERROR		0x800
+
 /**
  * After scanning the CQ array, the CQ head register needs to be updated.
  * Updating the register involves write memory barrier operations.
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (3 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data " Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
  2021-11-07 18:59   ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Thomas Monjalon
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

This patch add multi-process support for Kunpeng DMA devices.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 drivers/dma/hisilicon/hisi_dmadev.c | 21 +++++++++++++--------
 1 file changed, 13 insertions(+), 8 deletions(-)

diff --git a/drivers/dma/hisilicon/hisi_dmadev.c b/drivers/dma/hisilicon/hisi_dmadev.c
index d03967cae3..05066b4d0e 100644
--- a/drivers/dma/hisilicon/hisi_dmadev.c
+++ b/drivers/dma/hisilicon/hisi_dmadev.c
@@ -392,8 +392,10 @@ hisi_dma_stop(struct rte_dma_dev *dev)
 static int
 hisi_dma_close(struct rte_dma_dev *dev)
 {
-	/* The dmadev already stopped */
-	hisi_dma_free_iomem(dev->data->dev_private);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		/* The dmadev already stopped */
+		hisi_dma_free_iomem(dev->data->dev_private);
+	}
 	return 0;
 }
 
@@ -815,11 +817,13 @@ hisi_dma_create(struct rte_pci_device *pci_dev, uint8_t queue_id,
 	hw->cq_head_reg = hisi_dma_queue_regaddr(hw,
 						 HISI_DMA_QUEUE_CQ_HEAD_REG);
 
-	ret = hisi_dma_reset_hw(hw);
-	if (ret) {
-		HISI_DMA_LOG(ERR, "%s init device fail!", name);
-		(void)rte_dma_pmd_release(name);
-		return -EIO;
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY) {
+		ret = hisi_dma_reset_hw(hw);
+		if (ret) {
+			HISI_DMA_LOG(ERR, "%s init device fail!", name);
+			(void)rte_dma_pmd_release(name);
+			return -EIO;
+		}
 	}
 
 	dev->state = RTE_DMA_DEV_READY;
@@ -872,7 +876,8 @@ hisi_dma_probe(struct rte_pci_driver *pci_drv __rte_unused,
 		return ret;
 	HISI_DMA_LOG(DEBUG, "%s read PCI revision: 0x%x", name, revision);
 
-	hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
+	if (rte_eal_process_type() == RTE_PROC_PRIMARY)
+		hisi_dma_init_gbl(pci_dev->mem_resource[2].addr, revision);
 
 	for (i = 0; i < HISI_DMA_MAX_HW_QUEUES; i++) {
 		ret = hisi_dma_create(pci_dev, i, revision);
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (4 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process Chengwen Feng
@ 2021-11-02 12:37   ` Chengwen Feng
  2021-11-07 18:59   ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Thomas Monjalon
  6 siblings, 0 replies; 16+ messages in thread
From: Chengwen Feng @ 2021-11-02 12:37 UTC (permalink / raw)
  To: thomas; +Cc: dev

add Kunpeng DMA device ID to dmadev category.

Signed-off-by: Chengwen Feng <fengchengwen@huawei.com>
---
 usertools/dpdk-devbind.py | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/usertools/dpdk-devbind.py b/usertools/dpdk-devbind.py
index bb00f43702..a74a68ed82 100755
--- a/usertools/dpdk-devbind.py
+++ b/usertools/dpdk-devbind.py
@@ -68,10 +68,14 @@
 intel_ntb_icx = {'Class': '06', 'Vendor': '8086', 'Device': '347e',
                  'SVendor': None, 'SDevice': None}
 
+hisilicon_dma = {'Class': '08', 'Vendor': '19e5', 'Device': 'a122',
+                 'SVendor': None, 'SDevice': None}
+
 network_devices = [network_class, cavium_pkx, avp_vnic, ifpga_class]
 baseband_devices = [acceleration_class]
 crypto_devices = [encryption_class, intel_processor_class]
-dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx]
+dma_devices = [intel_idxd_spr, intel_ioat_bdw, intel_ioat_icx, intel_ioat_skx,
+               hisilicon_dma]
 eventdev_devices = [cavium_sso, cavium_tim, intel_dlb, octeontx2_sso]
 mempool_devices = [cavium_fpa, octeontx2_npa]
 compress_devices = [cavium_zip]
-- 
2.33.0


^ permalink raw reply related	[flat|nested] 16+ messages in thread

* Re: [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver
  2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
                     ` (5 preceding siblings ...)
  2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
@ 2021-11-07 18:59   ` Thomas Monjalon
  6 siblings, 0 replies; 16+ messages in thread
From: Thomas Monjalon @ 2021-11-07 18:59 UTC (permalink / raw)
  To: Chengwen Feng; +Cc: dev

02/11/2021 13:37, Chengwen Feng:
> This patch set add hisilicon DMA driver.
> 
> Chengwen Feng (6):
>   dma/hisilicon: add device probe and remove functions
>   dma/hisilicon: add dmadev instances create and destroy
>   dma/hisilicon: add control path functions
>   dma/hisilicon: add data path functions
>   dma/hisilicon: support multi-process
>   devbind: add Kunpeng DMA to dmadev category

Applied with few minor fixes, thanks.




^ permalink raw reply	[flat|nested] 16+ messages in thread

end of thread, other threads:[~2021-11-07 18:59 UTC | newest]

Thread overview: 16+ messages (download: mbox.gz follow: Atom feed
-- links below jump to the message on this page --
2021-10-30 10:36 [dpdk-dev] [PATCH 0/6] dma: add hisilicon DMA driver Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 3/6] dma/hisilicon: add control path functions Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 4/6] dma/hisilicon: add data " Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 5/6] dma/hisilicon: support multi-process Chengwen Feng
2021-10-30 10:36 ` [dpdk-dev] [PATCH 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
2021-11-02 12:37 ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 1/6] dma/hisilicon: add device probe and remove functions Chengwen Feng
2021-11-07 17:28     ` Thomas Monjalon
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 2/6] dma/hisilicon: add dmadev instances create and destroy Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 3/6] dma/hisilicon: add control path functions Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 4/6] dma/hisilicon: add data " Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 5/6] dma/hisilicon: support multi-process Chengwen Feng
2021-11-02 12:37   ` [dpdk-dev] [PATCH v2 6/6] devbind: add Kunpeng DMA to dmadev category Chengwen Feng
2021-11-07 18:59   ` [dpdk-dev] [PATCH v2 0/6] dma: add hisilicon DMA driver Thomas Monjalon

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.