[PATCH v3 0/2] introduce vdpa sample

[PATCH v3 0/2] introduce vdpa sample

[Xiaolong Ye]

Hi,

This patchset introduces vdpa sample to demonstrate the vDPA use case.

v3 changes:
* list cmd would show queue number and supported features of vdpa devices.
* address Xiao's review comments

v2 changes:

* fix a compilation error reported by Rosen
* improve create cmd in interactive mode and add two new cmds: list,
* quit
* add application documentation

Xiaolong Ye (2):
  vhost: introduce API to get vDPA device number
  examples/vdpa: introduce a new sample for vDPA

 MAINTAINERS                            |   2 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 115 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 458 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 lib/librte_vhost/rte_vdpa.h            |   3 +
 lib/librte_vhost/rte_vhost_version.map |   1 +
 lib/librte_vhost/vdpa.c                |   6 +
 10 files changed, 635 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

-- 
2.17.1

[PATCH v3 1/2] vhost: introduce API to get vDPA device number

[Xiaolong Ye]

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 lib/librte_vhost/rte_vdpa.h            | 3 +++
 lib/librte_vhost/rte_vhost_version.map | 1 +
 lib/librte_vhost/vdpa.c                | 6 ++++++
 3 files changed, 10 insertions(+)

diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
index 90465ca26..b8223e337 100644
--- a/lib/librte_vhost/rte_vdpa.h
+++ b/lib/librte_vhost/rte_vdpa.h
@@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
 struct rte_vdpa_device * __rte_experimental
 rte_vdpa_get_device(int did);
 
+/* Get current available vdpa device number */
+int __rte_experimental
+rte_vdpa_get_device_num(void);
 #endif /* _RTE_VDPA_H_ */
diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
index da220dd02..ae39b6e21 100644
--- a/lib/librte_vhost/rte_vhost_version.map
+++ b/lib/librte_vhost/rte_vhost_version.map
@@ -67,6 +67,7 @@ EXPERIMENTAL {
 	rte_vdpa_unregister_device;
 	rte_vdpa_find_device_id;
 	rte_vdpa_get_device;
+	rte_vdpa_get_device_num;
 	rte_vhost_driver_attach_vdpa_device;
 	rte_vhost_driver_detach_vdpa_device;
 	rte_vhost_driver_get_vdpa_device_id;
diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
index c82fd4370..c2c5dff1d 100644
--- a/lib/librte_vhost/vdpa.c
+++ b/lib/librte_vhost/vdpa.c
@@ -113,3 +113,9 @@ rte_vdpa_get_device(int did)
 
 	return vdpa_devices[did];
 }
+
+int
+rte_vdpa_get_device_num(void)
+{
+	return vdpa_device_num;
+}
-- 
2.17.1

[PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA

[Xiaolong Ye]

The vdpa sample application creates vhost-user sockets by using the
vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
virtio ring compatible devices to serve virtio driver directly to enable
datapath acceleration. As vDPA driver can help to set up vhost datapath,
this application doesn't need to launch dedicated worker threads for vhost
enqueue/dequeue operations.

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 MAINTAINERS                        |   2 +
 doc/guides/sample_app_ug/index.rst |   1 +
 doc/guides/sample_app_ug/vdpa.rst  | 115 ++++++++
 examples/Makefile                  |   2 +-
 examples/vdpa/Makefile             |  32 ++
 examples/vdpa/main.c               | 458 +++++++++++++++++++++++++++++
 examples/vdpa/meson.build          |  16 +
 7 files changed, 625 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

diff --git a/MAINTAINERS b/MAINTAINERS
index 5967c1dd3..5656f18e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
 F: examples/vhost_scsi/
 F: doc/guides/sample_app_ug/vhost_scsi.rst
 F: examples/vhost_crypto/
+F: examples/vdpa/
+F: doc/guides/sample_app_ug/vdpa.rst
 
 Vhost PMD
 M: Maxime Coquelin <maxime.coquelin@redhat.com>
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 5bedf4f6f..74b12af85 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -45,6 +45,7 @@ Sample Applications User Guides
     vhost
     vhost_scsi
     vhost_crypto
+    vdpa
     netmap_compatibility
     ip_pipeline
     test_pipeline
diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
new file mode 100644
index 000000000..44fe6736d
--- /dev/null
+++ b/doc/guides/sample_app_ug/vdpa.rst
@@ -0,0 +1,115 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+Vdpa Sample Application
+=======================
+
+The vdpa sample application creates vhost-user sockets by using the
+vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+virtio ring compatible devices to serve virtio driver directly to enable
+datapath acceleration. As vDPA driver can help to set up vhost datapath,
+this application doesn't need to launch dedicated worker threads for vhost
+enqueue/dequeue operations.
+
+Testing steps
+-------------
+
+This section shows the steps of how to start VMs with vDPA vhost-user
+backend and verify network connection & live migration.
+
+Build
+~~~~~
+
+To compile the sample application see :doc:`compiling`.
+
+The application is located in the ``vdpa`` sub-directory.
+
+Start the vdpa example
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+        ./vdpa [EAL options]  -- [--interactive|-i] or [--iface SOCKET_PATH]
+
+where
+
+* --iface specifies the path prefix of the UNIX domain socket file, e.g.
+  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
+  (n starts from 0).
+* --interactive means run the vdpa sample in interactive mode, currently 4
+  internal cmds are supported:
+
+  1. help: show help message
+  2. list: list all available vdpa devices
+  3. create: create a new vdpa port with socket file and vdpa device address
+  4. quit: unregister vhost driver and exit the application
+
+Take IFCVF driver for example:
+
+.. code-block:: console
+
+        ./vdpa --log-level=9 -c 0x6 -n 4 --socket-mem 1024,1024 \
+                -w 0000:06:00.2,vdpa=1 -w 0000:06:00.3,vdpa=1 \
+                -- --interactive
+
+.. note::
+    We need to bind vfio-pci to VFs before running vdpa sample.
+
+    * modprobe vfio-pci
+    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.2 06:00.3
+
+Then we can create 2 vdpa ports in interactive cmdline.
+
+.. code-block:: console
+
+        vdpa> list
+        device id       device address
+        0               0000:06:00.2
+        1               0000:06:00.3
+        vdpa> create /tmp/vdpa-socket0 0000:06:00.2
+        vdpa> create /tmp/vdpa-socket1 0000:06:00.3
+
+.. _vdpa_app_run_vm:
+
+Start the VMs
+~~~~~~~~~~~~~
+
+.. code-block:: console
+
+       qemu-system-x86_64 -cpu host -enable-kvm \
+       <snip>
+       -mem-prealloc \
+       -chardev socket,id=char0,path=<socket_file created in above steps> \
+       -netdev type=vhost-user,id=vdpa,chardev=char0 \
+       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
+
+After the VMs launches, we can login the VMs and configure the ip, verify the
+network connection via ping or netperf.
+
+.. note::
+    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
+
+Live Migration
+~~~~~~~~~~~~~~
+vDPA supports cross-backend live migration, user can migrate SW vhost backend
+VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
+the source host with SW vhost VM and B is the destination host with vDPA.
+
+1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
+   in migration-listen mode:
+
+.. code-block:: console
+
+        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
+
+2. Start the migration (on source host):
+
+.. code-block:: console
+
+        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
+
+3. Check the status (on source host):
+
+.. code-block:: console
+
+        A: (qemu) info migrate
diff --git a/examples/Makefile b/examples/Makefile
index 481720cb6..356fcb1cd 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
-DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
+DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
 ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
 endif
diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
new file mode 100644
index 000000000..42672a2bc
--- /dev/null
+++ b/examples/vdpa/Makefile
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
+$(info This application can only operate in a linuxapp environment, \
+please change the definition of the RTE_TARGET environment variable)
+all:
+else
+
+# binary name
+APP = vdpa
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -D_GNU_SOURCE
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
new file mode 100644
index 000000000..b190e67a4
--- /dev/null
+++ b/examples/vdpa/main.c
@@ -0,0 +1,458 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <getopt.h>
+#include <signal.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_ethdev.h>
+#include <rte_malloc.h>
+#include <rte_vhost.h>
+#include <rte_vdpa.h>
+#include <rte_pci.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#define MAX_PATH_LEN 128
+#define MAX_VDPA_SAMPLE_PORTS 1024
+#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
+
+struct vdpa_port {
+	char ifname[MAX_PATH_LEN];
+	int did;
+	int vid;
+	uint64_t flags;
+};
+
+struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
+
+static char iface[MAX_PATH_LEN];
+static int dev_total;
+static int devcnt;
+static int interactive;
+
+/* display usage */
+static void
+vdpa_usage(const char *prgname)
+{
+	printf("Usage: %s [EAL options] -- "
+				 "	--interactive|-i: run in interactive mode.\n"
+				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n",
+				 prgname);
+}
+
+static int
+parse_args(int argc, char **argv)
+{
+	static const char *short_option = "i";
+	static struct option long_option[] = {
+		{"iface", required_argument, NULL, 0},
+		{"interactive", no_argument, &interactive, 1},
+		{NULL, 0, 0, 0},
+	};
+	int opt, idx;
+	char *prgname = argv[0];
+
+	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
+			!= EOF) {
+		switch (opt) {
+		case 'i':
+			printf("Interactive-mode selected\n");
+			interactive = 1;
+			break;
+		/* long options */
+		case 0:
+			if (strncmp(long_option[idx].name, "iface",
+						MAX_PATH_LEN) == 0) {
+				strncpy(iface, optarg, MAX_PATH_LEN);
+				printf("iface %s\n", iface);
+			}
+			if (!strcmp(long_option[idx].name, "interactive")) {
+				printf("Interactive-mode selected\n");
+				interactive = 1;
+			}
+			break;
+
+		default:
+			vdpa_usage(prgname);
+			return -1;
+		}
+	}
+
+	if (iface[0] == '\0' && interactive == 0) {
+		vdpa_usage(prgname);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+data_init(void)
+{
+	dev_total = rte_vdpa_get_device_num();
+	if (dev_total <= 0) {
+		printf("No available vdpa device found\n");
+		return -1;
+	}
+	devcnt = 0;
+	interactive = 0;
+	memset(iface, 0, sizeof(iface));
+	memset(vports, 0, sizeof(vports));
+	return 0;
+}
+
+static int
+new_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
+			printf("\nnew port %s, did: %d\n",
+					ifname, vports[i].did);
+			vports[i].vid = vid;
+			break;
+		}
+	}
+
+	if (i >= MAX_VDPA_SAMPLE_PORTS)
+		return -1;
+
+	return 0;
+}
+
+static void
+destroy_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strcmp(ifname, vports[i].ifname) == 0) {
+			printf("\ndestroy port %s, did: %d\n",
+					ifname, vports[i].did);
+			break;
+		}
+	}
+}
+
+static const struct vhost_device_ops vdpa_sample_devops = {
+	.new_device = new_device,
+	.destroy_device = destroy_device,
+};
+
+static int
+start_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+	int did = vport->did;
+
+	if (access(socket_path, F_OK) != -1) {
+		RTE_LOG(ERR, VDPA,
+			"%s exists, please remove it or specify another file and try again.\n",
+			socket_path);
+		return -1;
+	}
+	ret = rte_vhost_driver_register(socket_path, vport->flags);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_callback_register(socket_path,
+			&vdpa_sample_devops);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver ops failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"attach vdpa device failed: %s\n",
+			socket_path);
+
+	if (rte_vhost_driver_start(socket_path) < 0)
+		rte_exit(EXIT_FAILURE,
+			"start vhost driver failed: %s\n",
+			socket_path);
+	return 0;
+}
+
+static void
+close_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+
+	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"attach vdpa device failed: %s\n",
+				socket_path);
+
+	ret = rte_vhost_driver_unregister(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"Fail to unregister vhost driver for %s.\n",
+				socket_path);
+}
+
+static void
+vdpa_sample_quit(void)
+{
+	int i;
+	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
+		if (vports[i].ifname[0] != '\0')
+			close_vdpa(&vports[i]);
+	}
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\nSignal %d received, preparing to exit...\n", signum);
+		vdpa_sample_quit();
+		exit(0);
+	}
+}
+
+/* interactive cmds */
+
+/* *** Help command with introduction. *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	cmdline_printf(
+		cl,
+		"\n"
+		"The following commands are currently available:\n\n"
+		"Control:\n"
+		"    help                                      : Show interactive instructions.\n"
+		"    list                                      : list all available vdpa devices.\n"
+		"    create <socket file> <vdev addr>          : create a new vdpa port.\n"
+		"    quit                                      : exit vdpa sample app.\n"
+	);
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show help",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/* *** List all available vdpa devices *** */
+struct cmd_list_result {
+	cmdline_fixed_string_t action;
+};
+
+static void cmd_list_vdpa_devices_parsed(
+		__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	uint32_t queue_num;
+	uint64_t features;
+	struct rte_vdpa_device *vdev;
+	struct rte_pci_addr addr;
+
+	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
+	for (did = 0; did < dev_total; did++) {
+		vdev = rte_vdpa_get_device(did);
+		if (!vdev)
+			continue;
+		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa queue number "
+				"for device id %d.\n", did);
+			continue;
+		}
+		if (vdev->ops->get_features(did, &features) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa features "
+				"for device id %d.\n", did);
+			continue;
+		}
+		addr = vdev->addr.pci_addr;
+		cmdline_printf(cl, "%d\t\t"PCI_PRI_FMT"\t%d\t\t0x%lx\n", did,
+				addr.domain, addr.bus, addr.devid,
+				addr.function, queue_num, features);
+	}
+}
+
+cmdline_parse_token_string_t cmd_action_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
+
+cmdline_parse_inst_t cmd_list_vdpa_devices = {
+	.f = cmd_list_vdpa_devices_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "list all available vdpa devices",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_action_list,
+		NULL,
+	},
+};
+
+/* *** Create new vdpa port *** */
+struct cmd_create_result {
+	cmdline_fixed_string_t action;
+	cmdline_fixed_string_t socket_path;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_create_vdpa_port_parsed(void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	struct cmd_create_result *res = parsed_result;
+	struct rte_vdpa_dev_addr addr;
+
+	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
+	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
+		cmdline_printf(cl, "Unable to parse the given bdf.\n");
+		return;
+	}
+	addr.type = PCI_ADDR;
+	did = rte_vdpa_find_device_id(&addr);
+	if (did < 0) {
+		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
+		return;
+	}
+
+	vports[devcnt].did = did;
+
+	if (start_vdpa(&vports[devcnt]) == 0)
+		devcnt++;
+}
+
+cmdline_parse_token_string_t cmd_action_create =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
+cmdline_parse_token_string_t cmd_socket_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
+cmdline_parse_token_string_t cmd_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_create_vdpa_port = {
+	.f = cmd_create_vdpa_port_parsed,  /* function to call */
+	.data = NULL,      /* 2nd arg of func */
+	.help_str = "create a new vdpa port",
+	.tokens = {        /* token list, NULL terminated */
+		(void *)&cmd_action_create,
+		(void *)&cmd_socket_path,
+		(void *)&cmd_bdf,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	vdpa_sample_quit();
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "quit: exit application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
+	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	NULL,
+};
+
+int
+main(int argc, char *argv[])
+{
+	char ch;
+	int i;
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "eal init failed\n");
+	argc -= ret;
+	argv += ret;
+
+	data_init();
+
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	ret = parse_args(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "invalid argument\n");
+
+	if (interactive == 1) {
+		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
+		if (cl == NULL)
+			rte_panic("Cannot create cmdline instance\n");
+		cmdline_interact(cl);
+		cmdline_stdin_exit(cl);
+	} else {
+		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
+				i++) {
+			vports[i].did = i;
+			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
+					iface, i);
+
+			start_vdpa(&vports[i]);
+		}
+
+		printf("enter \'q\' to quit\n");
+		while (scanf("%c", &ch)) {
+			if (ch == 'q')
+				break;
+			while (ch != '\n') {
+				if (scanf("%c", &ch))
+					printf("%c", ch);
+			}
+			printf("enter \'q\' to quit\n");
+		}
+	}
+
+	return 0;
+}
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
new file mode 100644
index 000000000..2e38a069f
--- /dev/null
+++ b/examples/vdpa/meson.build
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'vhost'
+allow_experimental_apis = true
+sources = files(
+	'main.c'
+)
\ No newline at end of file
-- 
2.17.1

Re: [PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA

[Wang, Xiao W]

Hi Xiaolong,

> -----Original Message-----
> From: Ye, Xiaolong
> Sent: Friday, September 21, 2018 6:28 AM
> To: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Bie,
> Tiwei <tiwei.bie@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>
> Cc: Wang, Xiao W <xiao.w.wang@intel.com>; Rami Rosen
> <roszenrami@gmail.com>; Wang, Haiyue <haiyue.wang@intel.com>; Ye,
> Xiaolong <xiaolong.ye@intel.com>
> Subject: [PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA
> 
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>  MAINTAINERS                        |   2 +
>  doc/guides/sample_app_ug/index.rst |   1 +
>  doc/guides/sample_app_ug/vdpa.rst  | 115 ++++++++
>  examples/Makefile                  |   2 +-
>  examples/vdpa/Makefile             |  32 ++
>  examples/vdpa/main.c               | 458 +++++++++++++++++++++++++++++
>  examples/vdpa/meson.build          |  16 +
>  7 files changed, 625 insertions(+), 1 deletion(-)
>  create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>  create mode 100644 examples/vdpa/Makefile
>  create mode 100644 examples/vdpa/main.c
>  create mode 100644 examples/vdpa/meson.build
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
>  F: examples/vhost_scsi/
>  F: doc/guides/sample_app_ug/vhost_scsi.rst
>  F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
> 
>  Vhost PMD
>  M: Maxime Coquelin <maxime.coquelin@redhat.com>
> diff --git a/doc/guides/sample_app_ug/index.rst
> b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
>      vhost
>      vhost_scsi
>      vhost_crypto
> +    vdpa
>      netmap_compatibility
>      ip_pipeline
>      test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst
> b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..44fe6736d
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,115 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps

[...]

> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> +        vdpa> list
> +        device id       device address
> +        0               0000:06:00.2
> +        1               0000:06:00.3

The features and queue numbers of each vDPA device could also be shown now, the doc should reflect this.

> +        vdpa> create /tmp/vdpa-socket0 0000:06:00.2
> +        vdpa> create /tmp/vdpa-socket1 0000:06:00.3
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 -cpu host -enable-kvm \
> +       <snip>
> +       -mem-prealloc \
> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> +       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> +    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
> +
> +Live Migration
> +~~~~~~~~~~~~~~
> +vDPA supports cross-backend live migration, user can migrate SW vhost
> backend
> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume
> A is
> +the source host with SW vhost VM and B is the destination host with vDPA.
> +
> +1. Start vdpa sample and launch a VM with exact same parameters as the VM
> on A,
> +   in migration-listen mode:
> +
> +.. code-block:: console
> +
> +        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
> +
> +2. Start the migration (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
> +
> +3. Check the status (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) info migrate
> diff --git a/examples/Makefile b/examples/Makefile
> index 481720cb6..356fcb1cd 100644
> --- a/examples/Makefile
> +++ b/examples/Makefile
> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
>  DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
>  endif
>  DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
>  ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
>  DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
>  endif
> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
> new file mode 100644
> index 000000000..42672a2bc
> --- /dev/null
> +++ b/examples/vdpa/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +ifeq ($(RTE_SDK),)
> +$(error "Please define RTE_SDK environment variable")
> +endif
> +
> +# Default target, can be overridden by command line or environment
> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
> +$(info This application can only operate in a linuxapp environment, \
> +please change the definition of the RTE_TARGET environment variable)
> +all:
> +else
> +
> +# binary name
> +APP = vdpa
> +
> +# all source are stored in SRCS-y
> +SRCS-y := main.c
> +
> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -D_GNU_SOURCE
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +
> +include $(RTE_SDK)/mk/rte.extapp.mk
> +

[...]

> +
> +static int
> +new_device(int vid)
> +{
> +	char ifname[MAX_PATH_LEN];
> +	int i;
> +
> +	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
> +	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
> +		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
> +			printf("\nnew port %s, did: %d\n",
> +					ifname, vports[i].did);
> +			vports[i].vid = vid;
> +			break;
> +		}
> +	}
> +
> +	if (i >= MAX_VDPA_SAMPLE_PORTS)
> +		return -1;
> +
> +	return 0;
> +}
> +
> +static void
> +destroy_device(int vid)
> +{
> +	char ifname[MAX_PATH_LEN];
> +	int i;
> +
> +	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
> +	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
> +		if (strcmp(ifname, vports[i].ifname) == 0) {
> +			printf("\ndestroy port %s, did: %d\n",
> +					ifname, vports[i].did);
> +			break;
> +		}
> +	}
> +}
> +
> +static const struct vhost_device_ops vdpa_sample_devops = {
> +	.new_device = new_device,
> +	.destroy_device = destroy_device,
> +};
> +
> +static int
> +start_vdpa(struct vdpa_port *vport)
> +{
> +	int ret;
> +	char *socket_path = vport->ifname;
> +	int did = vport->did;
> +
> +	if (access(socket_path, F_OK) != -1) {
> +		RTE_LOG(ERR, VDPA,
> +			"%s exists, please remove it or specify another file and
> try again.\n",
> +			socket_path);
> +		return -1;
> +	}
> +	ret = rte_vhost_driver_register(socket_path, vport->flags);

It would be great if we can provide "client mode" option for this sample to create a client mode vdpa.

> +	if (ret != 0)
> +		rte_exit(EXIT_FAILURE,
> +			"register driver failed: %s\n",
> +			socket_path);
> +
> +	ret = rte_vhost_driver_callback_register(socket_path,
> +			&vdpa_sample_devops);
> +	if (ret != 0)
> +		rte_exit(EXIT_FAILURE,
> +			"register driver ops failed: %s\n",
> +			socket_path);
> +
> +	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
> +	if (ret != 0)
> +		rte_exit(EXIT_FAILURE,
> +			"attach vdpa device failed: %s\n",
> +			socket_path);
> +
> +	if (rte_vhost_driver_start(socket_path) < 0)
> +		rte_exit(EXIT_FAILURE,
> +			"start vhost driver failed: %s\n",
> +			socket_path);
> +	return 0;
> +}
> +
> +static void
> +close_vdpa(struct vdpa_port *vport)
> +{
> +	int ret;
> +	char *socket_path = vport->ifname;
> +
> +	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
> +	if (ret != 0)
> +		RTE_LOG(ERR, VDPA,
> +				"attach vdpa device failed: %s\n",
> +				socket_path);
> +
> +	ret = rte_vhost_driver_unregister(socket_path);
> +	if (ret != 0)
> +		RTE_LOG(ERR, VDPA,
> +				"Fail to unregister vhost driver for %s.\n",
> +				socket_path);
> +}
> +
> +static void
> +vdpa_sample_quit(void)
> +{
> +	int i;
> +	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
> +		if (vports[i].ifname[0] != '\0')
> +			close_vdpa(&vports[i]);
> +	}
> +}
> +
> +static void
> +signal_handler(int signum)
> +{
> +	if (signum == SIGINT || signum == SIGTERM) {
> +		printf("\nSignal %d received, preparing to exit...\n", signum);
> +		vdpa_sample_quit();
> +		exit(0);
> +	}
> +}
> +
> +/* interactive cmds */
> +
> +/* *** Help command with introduction. *** */
> +struct cmd_help_result {
> +	cmdline_fixed_string_t help;
> +};
> +
> +static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	cmdline_printf(
> +		cl,
> +		"\n"
> +		"The following commands are currently available:\n\n"
> +		"Control:\n"
> +		"    help                                      : Show interactive instructions.\n"
> +		"    list                                      : list all available vdpa devices.\n"
> +		"    create <socket file> <vdev addr>          : create a new vdpa
> port.\n"
> +		"    quit                                      : exit vdpa sample app.\n"
> +	);
> +}
> +
> +cmdline_parse_token_string_t cmd_help_help =
> +	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
> +
> +cmdline_parse_inst_t cmd_help = {
> +	.f = cmd_help_parsed,
> +	.data = NULL,
> +	.help_str = "show help",
> +	.tokens = {
> +		(void *)&cmd_help_help,
> +		NULL,
> +	},
> +};
> +
> +/* *** List all available vdpa devices *** */
> +struct cmd_list_result {
> +	cmdline_fixed_string_t action;
> +};
> +
> +static void cmd_list_vdpa_devices_parsed(
> +		__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	int did;
> +	uint32_t queue_num;
> +	uint64_t features;
> +	struct rte_vdpa_device *vdev;
> +	struct rte_pci_addr addr;
> +
> +	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported
> features\n");
> +	for (did = 0; did < dev_total; did++) {
> +		vdev = rte_vdpa_get_device(did);
> +		if (!vdev)
> +			continue;
> +		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa queue number "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		if (vdev->ops->get_features(did, &features) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa features "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		addr = vdev->addr.pci_addr;
> +		cmdline_printf(cl, "%d\t\t"PCI_PRI_FMT"\t%d\t\t0x%lx\n", did,
> +				addr.domain, addr.bus, addr.devid,
> +				addr.function, queue_num, features);

Use "%PRIu32"for queue_num, and use "PRIx64" instead of "%lx", otherwise you may have compile error on 32-bit machine.

> +	}
> +}
> +
> +cmdline_parse_token_string_t cmd_action_list =
> +	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
> +
> +cmdline_parse_inst_t cmd_list_vdpa_devices = {
> +	.f = cmd_list_vdpa_devices_parsed,  /* function to call */
> +	.data = NULL,      /* 2nd arg of func */

This comment doesn't provide much info.
Remove the comment for all the cmdline_parse_inst_t instances to keep alignment.

> +	.help_str = "list all available vdpa devices",
> +	.tokens = {        /* token list, NULL terminated */
> +		(void *)&cmd_action_list,
> +		NULL,
> +	},
> +};
> +
> +/* *** Create new vdpa port *** */
> +struct cmd_create_result {
> +	cmdline_fixed_string_t action;
> +	cmdline_fixed_string_t socket_path;
> +	cmdline_fixed_string_t bdf;
> +};
> +
> +static void cmd_create_vdpa_port_parsed(void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	int did;
> +	struct cmd_create_result *res = parsed_result;
> +	struct rte_vdpa_dev_addr addr;
> +
> +	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
> +	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
> +		cmdline_printf(cl, "Unable to parse the given bdf.\n");
> +		return;
> +	}
> +	addr.type = PCI_ADDR;
> +	did = rte_vdpa_find_device_id(&addr);
> +	if (did < 0) {
> +		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
> +		return;
> +	}
> +
> +	vports[devcnt].did = did;
> +
> +	if (start_vdpa(&vports[devcnt]) == 0)
> +		devcnt++;
> +}
> +
> +cmdline_parse_token_string_t cmd_action_create =
> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action,
> "create");
> +cmdline_parse_token_string_t cmd_socket_path =
> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path,
> NULL);
> +cmdline_parse_token_string_t cmd_bdf =
> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
> +
> +cmdline_parse_inst_t cmd_create_vdpa_port = {
> +	.f = cmd_create_vdpa_port_parsed,  /* function to call */
> +	.data = NULL,      /* 2nd arg of func */
> +	.help_str = "create a new vdpa port",
> +	.tokens = {        /* token list, NULL terminated */
> +		(void *)&cmd_action_create,
> +		(void *)&cmd_socket_path,
> +		(void *)&cmd_bdf,
> +		NULL,
> +	},
> +};
> +
> +/* *** QUIT *** */
> +struct cmd_quit_result {
> +	cmdline_fixed_string_t quit;
> +};
> +
> +static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	vdpa_sample_quit();
> +	cmdline_quit(cl);
> +}
> +
> +cmdline_parse_token_string_t cmd_quit_quit =
> +	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
> +
> +cmdline_parse_inst_t cmd_quit = {
> +	.f = cmd_quit_parsed,
> +	.data = NULL,
> +	.help_str = "quit: exit application",
> +	.tokens = {
> +		(void *)&cmd_quit_quit,
> +		NULL,
> +	},
> +};
> +cmdline_parse_ctx_t main_ctx[] = {
> +	(cmdline_parse_inst_t *)&cmd_help,
> +	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
> +	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
> +	(cmdline_parse_inst_t *)&cmd_quit,
> +	NULL,
> +};
> +
> +int
> +main(int argc, char *argv[])
> +{
> +	char ch;
> +	int i;
> +	int ret;
> +	struct cmdline *cl;
> +
> +	ret = rte_eal_init(argc, argv);
> +	if (ret < 0)
> +		rte_exit(EXIT_FAILURE, "eal init failed\n");
> +	argc -= ret;
> +	argv += ret;
> +
> +	data_init();
> +
> +	signal(SIGINT, signal_handler);
> +	signal(SIGTERM, signal_handler);
> +
> +	ret = parse_args(argc, argv);
> +	if (ret < 0)
> +		rte_exit(EXIT_FAILURE, "invalid argument\n");
> +
> +	if (interactive == 1) {
> +		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
> +		if (cl == NULL)
> +			rte_panic("Cannot create cmdline instance\n");
> +		cmdline_interact(cl);
> +		cmdline_stdin_exit(cl);
> +	} else {
> +		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
> +				i++) {
> +			vports[i].did = i;
> +			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
> +					iface, i);
> +
> +			start_vdpa(&vports[i]);
> +		}
> +
> +		printf("enter \'q\' to quit\n");
> +		while (scanf("%c", &ch)) {
> +			if (ch == 'q')
> +				break;
> +			while (ch != '\n') {
> +				if (scanf("%c", &ch))
> +					printf("%c", ch);
> +			}
> +			printf("enter \'q\' to quit\n");
> +		}

Do vdpa_sample_quit here to stop the vhost session and delete the vhost sockets.

> +	}
> +
> +	return 0;
> +}
> diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
> new file mode 100644
> index 000000000..2e38a069f
> --- /dev/null
> +++ b/examples/vdpa/meson.build
> @@ -0,0 +1,16 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +# meson file, for building this example as part of a main DPDK build.
> +#
> +# To build this example as a standalone application with an already-installed
> +# DPDK instance, use 'make'
> +
> +if host_machine.system() != 'linux'
> +	build = false
> +endif
> +deps += 'vhost'
> +allow_experimental_apis = true
> +sources = files(
> +	'main.c'
> +)
> \ No newline at end of file
> --
> 2.17.1

Do not forget to add this into 18.11 release_note.

BRs,
Xiao

Re: [PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

On 09/21, Wang, Xiao W wrote:
>Hi Xiaolong,
>
>> -----Original Message-----
>> From: Ye, Xiaolong
>> Sent: Friday, September 21, 2018 6:28 AM
>> To: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Bie,
>> Tiwei <tiwei.bie@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>
>> Cc: Wang, Xiao W <xiao.w.wang@intel.com>; Rami Rosen
>> <roszenrami@gmail.com>; Wang, Haiyue <haiyue.wang@intel.com>; Ye,
>> Xiaolong <xiaolong.ye@intel.com>
>> Subject: [PATCH v3 2/2] examples/vdpa: introduce a new sample for vDPA
>> 
>> The vdpa sample application creates vhost-user sockets by using the
>> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
>> virtio ring compatible devices to serve virtio driver directly to enable
>> datapath acceleration. As vDPA driver can help to set up vhost datapath,
>> this application doesn't need to launch dedicated worker threads for vhost
>> enqueue/dequeue operations.
>> 
>> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> ---
>>  MAINTAINERS                        |   2 +
>>  doc/guides/sample_app_ug/index.rst |   1 +
>>  doc/guides/sample_app_ug/vdpa.rst  | 115 ++++++++
>>  examples/Makefile                  |   2 +-
>>  examples/vdpa/Makefile             |  32 ++
>>  examples/vdpa/main.c               | 458 +++++++++++++++++++++++++++++
>>  examples/vdpa/meson.build          |  16 +
>>  7 files changed, 625 insertions(+), 1 deletion(-)
>>  create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>>  create mode 100644 examples/vdpa/Makefile
>>  create mode 100644 examples/vdpa/main.c
>>  create mode 100644 examples/vdpa/meson.build
>> 
>> diff --git a/MAINTAINERS b/MAINTAINERS
>> index 5967c1dd3..5656f18e8 100644
>> --- a/MAINTAINERS
>> +++ b/MAINTAINERS
>> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
>>  F: examples/vhost_scsi/
>>  F: doc/guides/sample_app_ug/vhost_scsi.rst
>>  F: examples/vhost_crypto/
>> +F: examples/vdpa/
>> +F: doc/guides/sample_app_ug/vdpa.rst
>> 
>>  Vhost PMD
>>  M: Maxime Coquelin <maxime.coquelin@redhat.com>
>> diff --git a/doc/guides/sample_app_ug/index.rst
>> b/doc/guides/sample_app_ug/index.rst
>> index 5bedf4f6f..74b12af85 100644
>> --- a/doc/guides/sample_app_ug/index.rst
>> +++ b/doc/guides/sample_app_ug/index.rst
>> @@ -45,6 +45,7 @@ Sample Applications User Guides
>>      vhost
>>      vhost_scsi
>>      vhost_crypto
>> +    vdpa
>>      netmap_compatibility
>>      ip_pipeline
>>      test_pipeline
>> diff --git a/doc/guides/sample_app_ug/vdpa.rst
>> b/doc/guides/sample_app_ug/vdpa.rst
>> new file mode 100644
>> index 000000000..44fe6736d
>> --- /dev/null
>> +++ b/doc/guides/sample_app_ug/vdpa.rst
>> @@ -0,0 +1,115 @@
>> +..  SPDX-License-Identifier: BSD-3-Clause
>> +    Copyright(c) 2018 Intel Corporation.
>> +
>> +Vdpa Sample Application
>> +=======================
>> +
>> +The vdpa sample application creates vhost-user sockets by using the
>> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
>> +virtio ring compatible devices to serve virtio driver directly to enable
>> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
>> +this application doesn't need to launch dedicated worker threads for vhost
>> +enqueue/dequeue operations.
>> +
>> +Testing steps
>
>[...]
>
>> +
>> +Then we can create 2 vdpa ports in interactive cmdline.
>> +
>> +.. code-block:: console
>> +
>> +        vdpa> list
>> +        device id       device address
>> +        0               0000:06:00.2
>> +        1               0000:06:00.3
>
>The features and queue numbers of each vDPA device could also be shown now, the doc should reflect this.

Got it.

>
>> +        vdpa> create /tmp/vdpa-socket0 0000:06:00.2
>> +        vdpa> create /tmp/vdpa-socket1 0000:06:00.3
>> +
>> +.. _vdpa_app_run_vm:
>> +
>> +Start the VMs
>> +~~~~~~~~~~~~~
>> +
>> +.. code-block:: console
>> +
>> +       qemu-system-x86_64 -cpu host -enable-kvm \
>> +       <snip>
>> +       -mem-prealloc \
>> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
>> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
>> +       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
>> +
>> +After the VMs launches, we can login the VMs and configure the ip, verify the
>> +network connection via ping or netperf.
>> +
>> +.. note::
>> +    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
>> +
>> +Live Migration
>> +~~~~~~~~~~~~~~
>> +vDPA supports cross-backend live migration, user can migrate SW vhost
>> backend
>> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume
>> A is
>> +the source host with SW vhost VM and B is the destination host with vDPA.
>> +
>> +1. Start vdpa sample and launch a VM with exact same parameters as the VM
>> on A,
>> +   in migration-listen mode:
>> +
>> +.. code-block:: console
>> +
>> +        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
>> +
>> +2. Start the migration (on source host):
>> +
>> +.. code-block:: console
>> +
>> +        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
>> +
>> +3. Check the status (on source host):
>> +
>> +.. code-block:: console
>> +
>> +        A: (qemu) info migrate
>> diff --git a/examples/Makefile b/examples/Makefile
>> index 481720cb6..356fcb1cd 100644
>> --- a/examples/Makefile
>> +++ b/examples/Makefile
>> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
>>  DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
>>  endif
>>  DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
>> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
>> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
>>  ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
>>  DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
>>  endif
>> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
>> new file mode 100644
>> index 000000000..42672a2bc
>> --- /dev/null
>> +++ b/examples/vdpa/Makefile
>> @@ -0,0 +1,32 @@
>> +# SPDX-License-Identifier: BSD-3-Clause
>> +# Copyright(c) 2018 Intel Corporation
>> +
>> +ifeq ($(RTE_SDK),)
>> +$(error "Please define RTE_SDK environment variable")
>> +endif
>> +
>> +# Default target, can be overridden by command line or environment
>> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
>> +
>> +include $(RTE_SDK)/mk/rte.vars.mk
>> +
>> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
>> +$(info This application can only operate in a linuxapp environment, \
>> +please change the definition of the RTE_TARGET environment variable)
>> +all:
>> +else
>> +
>> +# binary name
>> +APP = vdpa
>> +
>> +# all source are stored in SRCS-y
>> +SRCS-y := main.c
>> +
>> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
>> +CFLAGS += $(WERROR_FLAGS)
>> +CFLAGS += -D_GNU_SOURCE
>> +CFLAGS += -DALLOW_EXPERIMENTAL_API
>> +
>> +include $(RTE_SDK)/mk/rte.extapp.mk
>> +
>
>[...]
>
>> +
>> +static int
>> +new_device(int vid)
>> +{
>> +	char ifname[MAX_PATH_LEN];
>> +	int i;
>> +
>> +	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
>> +	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
>> +		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
>> +			printf("\nnew port %s, did: %d\n",
>> +					ifname, vports[i].did);
>> +			vports[i].vid = vid;
>> +			break;
>> +		}
>> +	}
>> +
>> +	if (i >= MAX_VDPA_SAMPLE_PORTS)
>> +		return -1;
>> +
>> +	return 0;
>> +}
>> +
>> +static void
>> +destroy_device(int vid)
>> +{
>> +	char ifname[MAX_PATH_LEN];
>> +	int i;
>> +
>> +	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
>> +	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
>> +		if (strcmp(ifname, vports[i].ifname) == 0) {
>> +			printf("\ndestroy port %s, did: %d\n",
>> +					ifname, vports[i].did);
>> +			break;
>> +		}
>> +	}
>> +}
>> +
>> +static const struct vhost_device_ops vdpa_sample_devops = {
>> +	.new_device = new_device,
>> +	.destroy_device = destroy_device,
>> +};
>> +
>> +static int
>> +start_vdpa(struct vdpa_port *vport)
>> +{
>> +	int ret;
>> +	char *socket_path = vport->ifname;
>> +	int did = vport->did;
>> +
>> +	if (access(socket_path, F_OK) != -1) {
>> +		RTE_LOG(ERR, VDPA,
>> +			"%s exists, please remove it or specify another file and
>> try again.\n",
>> +			socket_path);
>> +		return -1;
>> +	}
>> +	ret = rte_vhost_driver_register(socket_path, vport->flags);
>
>It would be great if we can provide "client mode" option for this sample to create a client mode vdpa.

Sure, I'll add client mode support.

>
>> +	if (ret != 0)
>> +		rte_exit(EXIT_FAILURE,
>> +			"register driver failed: %s\n",
>> +			socket_path);
>> +
>> +	ret = rte_vhost_driver_callback_register(socket_path,
>> +			&vdpa_sample_devops);
>> +	if (ret != 0)
>> +		rte_exit(EXIT_FAILURE,
>> +			"register driver ops failed: %s\n",
>> +			socket_path);
>> +
>> +	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
>> +	if (ret != 0)
>> +		rte_exit(EXIT_FAILURE,
>> +			"attach vdpa device failed: %s\n",
>> +			socket_path);
>> +
>> +	if (rte_vhost_driver_start(socket_path) < 0)
>> +		rte_exit(EXIT_FAILURE,
>> +			"start vhost driver failed: %s\n",
>> +			socket_path);
>> +	return 0;
>> +}
>> +
>> +static void
>> +close_vdpa(struct vdpa_port *vport)
>> +{
>> +	int ret;
>> +	char *socket_path = vport->ifname;
>> +
>> +	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
>> +	if (ret != 0)
>> +		RTE_LOG(ERR, VDPA,
>> +				"attach vdpa device failed: %s\n",
>> +				socket_path);
>> +
>> +	ret = rte_vhost_driver_unregister(socket_path);
>> +	if (ret != 0)
>> +		RTE_LOG(ERR, VDPA,
>> +				"Fail to unregister vhost driver for %s.\n",
>> +				socket_path);
>> +}
>> +
>> +static void
>> +vdpa_sample_quit(void)
>> +{
>> +	int i;
>> +	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
>> +		if (vports[i].ifname[0] != '\0')
>> +			close_vdpa(&vports[i]);
>> +	}
>> +}
>> +
>> +static void
>> +signal_handler(int signum)
>> +{
>> +	if (signum == SIGINT || signum == SIGTERM) {
>> +		printf("\nSignal %d received, preparing to exit...\n", signum);
>> +		vdpa_sample_quit();
>> +		exit(0);
>> +	}
>> +}
>> +
>> +/* interactive cmds */
>> +
>> +/* *** Help command with introduction. *** */
>> +struct cmd_help_result {
>> +	cmdline_fixed_string_t help;
>> +};
>> +
>> +static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
>> +		struct cmdline *cl,
>> +		__attribute__((unused)) void *data)
>> +{
>> +	cmdline_printf(
>> +		cl,
>> +		"\n"
>> +		"The following commands are currently available:\n\n"
>> +		"Control:\n"
>> +		"    help                                      : Show interactive instructions.\n"
>> +		"    list                                      : list all available vdpa devices.\n"
>> +		"    create <socket file> <vdev addr>          : create a new vdpa
>> port.\n"
>> +		"    quit                                      : exit vdpa sample app.\n"
>> +	);
>> +}
>> +
>> +cmdline_parse_token_string_t cmd_help_help =
>> +	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
>> +
>> +cmdline_parse_inst_t cmd_help = {
>> +	.f = cmd_help_parsed,
>> +	.data = NULL,
>> +	.help_str = "show help",
>> +	.tokens = {
>> +		(void *)&cmd_help_help,
>> +		NULL,
>> +	},
>> +};
>> +
>> +/* *** List all available vdpa devices *** */
>> +struct cmd_list_result {
>> +	cmdline_fixed_string_t action;
>> +};
>> +
>> +static void cmd_list_vdpa_devices_parsed(
>> +		__attribute__((unused)) void *parsed_result,
>> +		struct cmdline *cl,
>> +		__attribute__((unused)) void *data)
>> +{
>> +	int did;
>> +	uint32_t queue_num;
>> +	uint64_t features;
>> +	struct rte_vdpa_device *vdev;
>> +	struct rte_pci_addr addr;
>> +
>> +	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported
>> features\n");
>> +	for (did = 0; did < dev_total; did++) {
>> +		vdev = rte_vdpa_get_device(did);
>> +		if (!vdev)
>> +			continue;
>> +		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
>> +			RTE_LOG(ERR, VDPA,
>> +				"failed to get vdpa queue number "
>> +				"for device id %d.\n", did);
>> +			continue;
>> +		}
>> +		if (vdev->ops->get_features(did, &features) < 0) {
>> +			RTE_LOG(ERR, VDPA,
>> +				"failed to get vdpa features "
>> +				"for device id %d.\n", did);
>> +			continue;
>> +		}
>> +		addr = vdev->addr.pci_addr;
>> +		cmdline_printf(cl, "%d\t\t"PCI_PRI_FMT"\t%d\t\t0x%lx\n", did,
>> +				addr.domain, addr.bus, addr.devid,
>> +				addr.function, queue_num, features);
>
>Use "%PRIu32"for queue_num, and use "PRIx64" instead of "%lx", otherwise you may have compile error on 32-bit machine.

Got it.

>
>> +	}
>> +}
>> +
>> +cmdline_parse_token_string_t cmd_action_list =
>> +	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
>> +
>> +cmdline_parse_inst_t cmd_list_vdpa_devices = {
>> +	.f = cmd_list_vdpa_devices_parsed,  /* function to call */
>> +	.data = NULL,      /* 2nd arg of func */
>
>This comment doesn't provide much info.
>Remove the comment for all the cmdline_parse_inst_t instances to keep alignment.

Ok, I'll remove all useless comments.

>
>> +	.help_str = "list all available vdpa devices",
>> +	.tokens = {        /* token list, NULL terminated */
>> +		(void *)&cmd_action_list,
>> +		NULL,
>> +	},
>> +};
>> +
>> +/* *** Create new vdpa port *** */
>> +struct cmd_create_result {
>> +	cmdline_fixed_string_t action;
>> +	cmdline_fixed_string_t socket_path;
>> +	cmdline_fixed_string_t bdf;
>> +};
>> +
>> +static void cmd_create_vdpa_port_parsed(void *parsed_result,
>> +		struct cmdline *cl,
>> +		__attribute__((unused)) void *data)
>> +{
>> +	int did;
>> +	struct cmd_create_result *res = parsed_result;
>> +	struct rte_vdpa_dev_addr addr;
>> +
>> +	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
>> +	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
>> +		cmdline_printf(cl, "Unable to parse the given bdf.\n");
>> +		return;
>> +	}
>> +	addr.type = PCI_ADDR;
>> +	did = rte_vdpa_find_device_id(&addr);
>> +	if (did < 0) {
>> +		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
>> +		return;
>> +	}
>> +
>> +	vports[devcnt].did = did;
>> +
>> +	if (start_vdpa(&vports[devcnt]) == 0)
>> +		devcnt++;
>> +}
>> +
>> +cmdline_parse_token_string_t cmd_action_create =
>> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action,
>> "create");
>> +cmdline_parse_token_string_t cmd_socket_path =
>> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path,
>> NULL);
>> +cmdline_parse_token_string_t cmd_bdf =
>> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
>> +
>> +cmdline_parse_inst_t cmd_create_vdpa_port = {
>> +	.f = cmd_create_vdpa_port_parsed,  /* function to call */
>> +	.data = NULL,      /* 2nd arg of func */
>> +	.help_str = "create a new vdpa port",
>> +	.tokens = {        /* token list, NULL terminated */
>> +		(void *)&cmd_action_create,
>> +		(void *)&cmd_socket_path,
>> +		(void *)&cmd_bdf,
>> +		NULL,
>> +	},
>> +};
>> +
>> +/* *** QUIT *** */
>> +struct cmd_quit_result {
>> +	cmdline_fixed_string_t quit;
>> +};
>> +
>> +static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
>> +		struct cmdline *cl,
>> +		__attribute__((unused)) void *data)
>> +{
>> +	vdpa_sample_quit();
>> +	cmdline_quit(cl);
>> +}
>> +
>> +cmdline_parse_token_string_t cmd_quit_quit =
>> +	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
>> +
>> +cmdline_parse_inst_t cmd_quit = {
>> +	.f = cmd_quit_parsed,
>> +	.data = NULL,
>> +	.help_str = "quit: exit application",
>> +	.tokens = {
>> +		(void *)&cmd_quit_quit,
>> +		NULL,
>> +	},
>> +};
>> +cmdline_parse_ctx_t main_ctx[] = {
>> +	(cmdline_parse_inst_t *)&cmd_help,
>> +	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
>> +	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
>> +	(cmdline_parse_inst_t *)&cmd_quit,
>> +	NULL,
>> +};
>> +
>> +int
>> +main(int argc, char *argv[])
>> +{
>> +	char ch;
>> +	int i;
>> +	int ret;
>> +	struct cmdline *cl;
>> +
>> +	ret = rte_eal_init(argc, argv);
>> +	if (ret < 0)
>> +		rte_exit(EXIT_FAILURE, "eal init failed\n");
>> +	argc -= ret;
>> +	argv += ret;
>> +
>> +	data_init();
>> +
>> +	signal(SIGINT, signal_handler);
>> +	signal(SIGTERM, signal_handler);
>> +
>> +	ret = parse_args(argc, argv);
>> +	if (ret < 0)
>> +		rte_exit(EXIT_FAILURE, "invalid argument\n");
>> +
>> +	if (interactive == 1) {
>> +		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
>> +		if (cl == NULL)
>> +			rte_panic("Cannot create cmdline instance\n");
>> +		cmdline_interact(cl);
>> +		cmdline_stdin_exit(cl);
>> +	} else {
>> +		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
>> +				i++) {
>> +			vports[i].did = i;
>> +			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
>> +					iface, i);
>> +
>> +			start_vdpa(&vports[i]);
>> +		}
>> +
>> +		printf("enter \'q\' to quit\n");
>> +		while (scanf("%c", &ch)) {
>> +			if (ch == 'q')
>> +				break;
>> +			while (ch != '\n') {
>> +				if (scanf("%c", &ch))
>> +					printf("%c", ch);
>> +			}
>> +			printf("enter \'q\' to quit\n");
>> +		}
>
>Do vdpa_sample_quit here to stop the vhost session and delete the vhost sockets.

It needs to call vdpa_sample_quit here.

>
>> +	}
>> +
>> +	return 0;
>> +}
>> diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
>> new file mode 100644
>> index 000000000..2e38a069f
>> --- /dev/null
>> +++ b/examples/vdpa/meson.build
>> @@ -0,0 +1,16 @@
>> +# SPDX-License-Identifier: BSD-3-Clause
>> +# Copyright(c) 2018 Intel Corporation
>> +
>> +# meson file, for building this example as part of a main DPDK build.
>> +#
>> +# To build this example as a standalone application with an already-installed
>> +# DPDK instance, use 'make'
>> +
>> +if host_machine.system() != 'linux'
>> +	build = false
>> +endif
>> +deps += 'vhost'
>> +allow_experimental_apis = true
>> +sources = files(
>> +	'main.c'
>> +)
>> \ No newline at end of file
>> --
>> 2.17.1
>
>Do not forget to add this into 18.11 release_note.

Ok, does this need to make a new separate commit?

Thanks,
Xiaolong
>
>BRs,
>Xiao
>

[PATCH v4 0/2] introduce vdpa sample

[Xiaolong Ye]

Hi,

This patchset introduces vdpa sample to demonstrate the vDPA use case.

v4 changes:
* add client mode support
* improve the format to list the vDPA device info and improve the vdpa.rst 
  accordingly
* remove some useless comments
* add introduction in 18.11 release note.


v3 changes:
* list cmd would show queue number and supported features of vdpa devices.
* address Xiao's review comments

v2 changes:

* fix a compilation error reported by Rosen
* improve create cmd in interactive mode and add two new cmds: list,
* quit
* add application documentation


Xiaolong Ye (2):
  vhost: introduce API to get vDPA device number
  examples/vdpa: introduce a new sample for vDPA

 MAINTAINERS                            |   2 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 115 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 458 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 lib/librte_vhost/rte_vdpa.h            |   3 +
 lib/librte_vhost/rte_vhost_version.map |   1 +
 lib/librte_vhost/vdpa.c                |   6 +
 10 files changed, 635 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

-- 
2.17.1

[PATCH v4 1/2] vhost: introduce API to get vDPA device number

[Xiaolong Ye]

It's used to get number of available registered vDPA devices.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 lib/librte_vhost/rte_vdpa.h            | 3 +++
 lib/librte_vhost/rte_vhost_version.map | 1 +
 lib/librte_vhost/vdpa.c                | 6 ++++++
 3 files changed, 10 insertions(+)

diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
index 90465ca26..b8223e337 100644
--- a/lib/librte_vhost/rte_vdpa.h
+++ b/lib/librte_vhost/rte_vdpa.h
@@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
 struct rte_vdpa_device * __rte_experimental
 rte_vdpa_get_device(int did);
 
+/* Get current available vdpa device number */
+int __rte_experimental
+rte_vdpa_get_device_num(void);
 #endif /* _RTE_VDPA_H_ */
diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
index da220dd02..ae39b6e21 100644
--- a/lib/librte_vhost/rte_vhost_version.map
+++ b/lib/librte_vhost/rte_vhost_version.map
@@ -67,6 +67,7 @@ EXPERIMENTAL {
 	rte_vdpa_unregister_device;
 	rte_vdpa_find_device_id;
 	rte_vdpa_get_device;
+	rte_vdpa_get_device_num;
 	rte_vhost_driver_attach_vdpa_device;
 	rte_vhost_driver_detach_vdpa_device;
 	rte_vhost_driver_get_vdpa_device_id;
diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
index c82fd4370..c2c5dff1d 100644
--- a/lib/librte_vhost/vdpa.c
+++ b/lib/librte_vhost/vdpa.c
@@ -113,3 +113,9 @@ rte_vdpa_get_device(int did)
 
 	return vdpa_devices[did];
 }
+
+int
+rte_vdpa_get_device_num(void)
+{
+	return vdpa_device_num;
+}
-- 
2.17.1

[PATCH v4 2/2] examples/vdpa: introduce a new sample for vDPA

[Xiaolong Ye]

The vdpa sample application creates vhost-user sockets by using the
vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
virtio ring compatible devices to serve virtio driver directly to enable
datapath acceleration. As vDPA driver can help to set up vhost datapath,
this application doesn't need to launch dedicated worker threads for vhost
enqueue/dequeue operations.

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 MAINTAINERS                            |   2 +
 doc/guides/rel_notes/release_18_11.rst |   8 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 118 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 466 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 8 files changed, 644 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

diff --git a/MAINTAINERS b/MAINTAINERS
index 5967c1dd3..5656f18e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
 F: examples/vhost_scsi/
 F: doc/guides/sample_app_ug/vhost_scsi.rst
 F: examples/vhost_crypto/
+F: examples/vdpa/
+F: doc/guides/sample_app_ug/vdpa.rst
 
 Vhost PMD
 M: Maxime Coquelin <maxime.coquelin@redhat.com>
diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index bc9b74ec4..dd53a9ecf 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -67,6 +67,14 @@ New Features
   SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
   vdev_netvsc, tap, and failsafe drivers combination.
 
+* **Add a new sample for vDPA**
+
+  The vdpa sample application creates vhost-user sockets by using the
+  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+  virtio ring compatible devices to serve virtio driver directly to enable
+  datapath acceleration. As vDPA driver can help to set up vhost datapath,
+  this application doesn't need to launch dedicated worker threads for vhost
+  enqueue/dequeue operations.
 
 API Changes
 -----------
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 5bedf4f6f..74b12af85 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -45,6 +45,7 @@ Sample Applications User Guides
     vhost
     vhost_scsi
     vhost_crypto
+    vdpa
     netmap_compatibility
     ip_pipeline
     test_pipeline
diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
new file mode 100644
index 000000000..d05728a37
--- /dev/null
+++ b/doc/guides/sample_app_ug/vdpa.rst
@@ -0,0 +1,118 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+Vdpa Sample Application
+=======================
+
+The vdpa sample application creates vhost-user sockets by using the
+vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+virtio ring compatible devices to serve virtio driver directly to enable
+datapath acceleration. As vDPA driver can help to set up vhost datapath,
+this application doesn't need to launch dedicated worker threads for vhost
+enqueue/dequeue operations.
+
+Testing steps
+-------------
+
+This section shows the steps of how to start VMs with vDPA vhost-user
+backend and verify network connection & live migration.
+
+Build
+~~~~~
+
+To compile the sample application see :doc:`compiling`.
+
+The application is located in the ``vdpa`` sub-directory.
+
+Start the vdpa example
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
+
+where
+
+* --client means running vdpa app in client mode, in the client mode, QEMU needs
+  to run as the server mode and take charge of socket file creation.
+* --iface specifies the path prefix of the UNIX domain socket file, e.g.
+  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
+  (n starts from 0).
+* --interactive means run the vdpa sample in interactive mode, currently 4
+  internal cmds are supported:
+
+  1. help: show help message
+  2. list: list all available vdpa devices
+  3. create: create a new vdpa port with socket file and vdpa device address
+  4. quit: unregister vhost driver and exit the application
+
+Take IFCVF driver for example:
+
+.. code-block:: console
+
+        ./vdpa --log-level=9 -c 0x6 -n 4 --socket-mem 1024,1024 \
+                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
+                -- --interactive
+
+.. note::
+    We need to bind vfio-pci to VFs before running vdpa sample.
+
+    * modprobe vfio-pci
+    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
+
+Then we can create 2 vdpa ports in interactive cmdline.
+
+.. code-block:: console
+
+        vdpa> list
+        device id       device address  queue num       supported features
+        0               0000:06:00.3    1               0x5572362272
+        1               0000:06:00.4    1               0x5572362272
+
+        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
+        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
+
+.. _vdpa_app_run_vm:
+
+Start the VMs
+~~~~~~~~~~~~~
+
+.. code-block:: console
+
+       qemu-system-x86_64 -cpu host -enable-kvm \
+       <snip>
+       -mem-prealloc \
+       -chardev socket,id=char0,path=<socket_file created in above steps> \
+       -netdev type=vhost-user,id=vdpa,chardev=char0 \
+       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
+
+After the VMs launches, we can login the VMs and configure the ip, verify the
+network connection via ping or netperf.
+
+.. note::
+    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
+
+Live Migration
+~~~~~~~~~~~~~~
+vDPA supports cross-backend live migration, user can migrate SW vhost backend
+VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
+the source host with SW vhost VM and B is the destination host with vDPA.
+
+1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
+   in migration-listen mode:
+
+.. code-block:: console
+
+        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
+
+2. Start the migration (on source host):
+
+.. code-block:: console
+
+        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
+
+3. Check the status (on source host):
+
+.. code-block:: console
+
+        A: (qemu) info migrate
diff --git a/examples/Makefile b/examples/Makefile
index 481720cb6..356fcb1cd 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
-DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
+DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
 ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
 endif
diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
new file mode 100644
index 000000000..42672a2bc
--- /dev/null
+++ b/examples/vdpa/Makefile
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
+$(info This application can only operate in a linuxapp environment, \
+please change the definition of the RTE_TARGET environment variable)
+all:
+else
+
+# binary name
+APP = vdpa
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -D_GNU_SOURCE
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
new file mode 100644
index 000000000..62c5ac90b
--- /dev/null
+++ b/examples/vdpa/main.c
@@ -0,0 +1,466 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <getopt.h>
+#include <signal.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_ethdev.h>
+#include <rte_malloc.h>
+#include <rte_vhost.h>
+#include <rte_vdpa.h>
+#include <rte_pci.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#define MAX_PATH_LEN 128
+#define MAX_VDPA_SAMPLE_PORTS 1024
+#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
+
+struct vdpa_port {
+	char ifname[MAX_PATH_LEN];
+	int did;
+	int vid;
+	uint64_t flags;
+};
+
+struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
+
+static char iface[MAX_PATH_LEN];
+static int dev_total;
+static int devcnt;
+static int interactive;
+static int client_mode;
+
+/* display usage */
+static void
+vdpa_usage(const char *prgname)
+{
+	printf("Usage: %s [EAL options] -- "
+				 "	--interactive|-i: run in interactive mode.\n"
+				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				 "	--client: register a vhost-user socket as client mode.\n",
+				 prgname);
+}
+
+static int
+parse_args(int argc, char **argv)
+{
+	static const char *short_option = "i";
+	static struct option long_option[] = {
+		{"iface", required_argument, NULL, 0},
+		{"interactive", no_argument, &interactive, 1},
+		{"client", no_argument, &client_mode, 1},
+		{NULL, 0, 0, 0},
+	};
+	int opt, idx;
+	char *prgname = argv[0];
+
+	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
+			!= EOF) {
+		switch (opt) {
+		case 'i':
+			printf("Interactive-mode selected\n");
+			interactive = 1;
+			break;
+		/* long options */
+		case 0:
+			if (strncmp(long_option[idx].name, "iface",
+						MAX_PATH_LEN) == 0) {
+				strncpy(iface, optarg, MAX_PATH_LEN);
+				printf("iface %s\n", iface);
+			}
+			if (!strcmp(long_option[idx].name, "interactive")) {
+				printf("Interactive-mode selected\n");
+				interactive = 1;
+			}
+			break;
+
+		default:
+			vdpa_usage(prgname);
+			return -1;
+		}
+	}
+
+	if (iface[0] == '\0' && interactive == 0) {
+		vdpa_usage(prgname);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+data_init(void)
+{
+	dev_total = rte_vdpa_get_device_num();
+	if (dev_total <= 0) {
+		printf("No available vdpa device found\n");
+		return -1;
+	}
+	devcnt = 0;
+	interactive = 0;
+	memset(iface, 0, sizeof(iface));
+	memset(vports, 0, sizeof(vports));
+	return 0;
+}
+
+static int
+new_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
+			printf("\nnew port %s, did: %d\n",
+					ifname, vports[i].did);
+			vports[i].vid = vid;
+			break;
+		}
+	}
+
+	if (i >= MAX_VDPA_SAMPLE_PORTS)
+		return -1;
+
+	return 0;
+}
+
+static void
+destroy_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strcmp(ifname, vports[i].ifname) == 0) {
+			printf("\ndestroy port %s, did: %d\n",
+					ifname, vports[i].did);
+			break;
+		}
+	}
+}
+
+static const struct vhost_device_ops vdpa_sample_devops = {
+	.new_device = new_device,
+	.destroy_device = destroy_device,
+};
+
+static int
+start_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+	int did = vport->did;
+
+	if (client_mode)
+		vport->flags |= RTE_VHOST_USER_CLIENT;
+
+	if (access(socket_path, F_OK) != -1 && !client_mode) {
+		RTE_LOG(ERR, VDPA,
+			"%s exists, please remove it or specify another file and try again.\n",
+			socket_path);
+		return -1;
+	}
+	ret = rte_vhost_driver_register(socket_path, vport->flags);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_callback_register(socket_path,
+			&vdpa_sample_devops);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver ops failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"attach vdpa device failed: %s\n",
+			socket_path);
+
+	if (rte_vhost_driver_start(socket_path) < 0)
+		rte_exit(EXIT_FAILURE,
+			"start vhost driver failed: %s\n",
+			socket_path);
+	return 0;
+}
+
+static void
+close_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+
+	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"detach vdpa device failed: %s\n",
+				socket_path);
+
+	ret = rte_vhost_driver_unregister(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"Fail to unregister vhost driver for %s.\n",
+				socket_path);
+}
+
+static void
+vdpa_sample_quit(void)
+{
+	int i;
+	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
+		if (vports[i].ifname[0] != '\0')
+			close_vdpa(&vports[i]);
+	}
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\nSignal %d received, preparing to exit...\n", signum);
+		vdpa_sample_quit();
+		exit(0);
+	}
+}
+
+/* interactive cmds */
+
+/* *** Help command with introduction. *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	cmdline_printf(
+		cl,
+		"\n"
+		"The following commands are currently available:\n\n"
+		"Control:\n"
+		"    help                                      : Show interactive instructions.\n"
+		"    list                                      : list all available vdpa devices.\n"
+		"    create <socket file> <vdev addr>          : create a new vdpa port.\n"
+		"    quit                                      : exit vdpa sample app.\n"
+	);
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show help",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/* *** List all available vdpa devices *** */
+struct cmd_list_result {
+	cmdline_fixed_string_t action;
+};
+
+static void cmd_list_vdpa_devices_parsed(
+		__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	uint32_t queue_num;
+	uint64_t features;
+	struct rte_vdpa_device *vdev;
+	struct rte_pci_addr addr;
+
+	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
+	for (did = 0; did < dev_total; did++) {
+		vdev = rte_vdpa_get_device(did);
+		if (!vdev)
+			continue;
+		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa queue number "
+				"for device id %d.\n", did);
+			continue;
+		}
+		if (vdev->ops->get_features(did, &features) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa features "
+				"for device id %d.\n", did);
+			continue;
+		}
+		addr = vdev->addr.pci_addr;
+		cmdline_printf(cl,
+			"%d\t\t"PCI_PRI_FMT"\t%"PRIu32"\t\t0x%"PRIu64"\n", did,
+			addr.domain, addr.bus, addr.devid,
+			addr.function, queue_num, features);
+	}
+}
+
+cmdline_parse_token_string_t cmd_action_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
+
+cmdline_parse_inst_t cmd_list_vdpa_devices = {
+	.f = cmd_list_vdpa_devices_parsed,
+	.data = NULL,
+	.help_str = "list all available vdpa devices",
+	.tokens = {
+		(void *)&cmd_action_list,
+		NULL,
+	},
+};
+
+/* *** Create new vdpa port *** */
+struct cmd_create_result {
+	cmdline_fixed_string_t action;
+	cmdline_fixed_string_t socket_path;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_create_vdpa_port_parsed(void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	struct cmd_create_result *res = parsed_result;
+	struct rte_vdpa_dev_addr addr;
+
+	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
+	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
+		cmdline_printf(cl, "Unable to parse the given bdf.\n");
+		return;
+	}
+	addr.type = PCI_ADDR;
+	did = rte_vdpa_find_device_id(&addr);
+	if (did < 0) {
+		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
+		return;
+	}
+
+	vports[devcnt].did = did;
+
+	if (start_vdpa(&vports[devcnt]) == 0)
+		devcnt++;
+}
+
+cmdline_parse_token_string_t cmd_action_create =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
+cmdline_parse_token_string_t cmd_socket_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
+cmdline_parse_token_string_t cmd_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_create_vdpa_port = {
+	.f = cmd_create_vdpa_port_parsed,
+	.data = NULL,
+	.help_str = "create a new vdpa port",
+	.tokens = {
+		(void *)&cmd_action_create,
+		(void *)&cmd_socket_path,
+		(void *)&cmd_bdf,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	vdpa_sample_quit();
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "quit: exit application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
+	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	NULL,
+};
+
+int
+main(int argc, char *argv[])
+{
+	char ch;
+	int i;
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "eal init failed\n");
+	argc -= ret;
+	argv += ret;
+
+	data_init();
+
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	ret = parse_args(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "invalid argument\n");
+
+	if (interactive == 1) {
+		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
+		if (cl == NULL)
+			rte_panic("Cannot create cmdline instance\n");
+		cmdline_interact(cl);
+		cmdline_stdin_exit(cl);
+	} else {
+		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
+				i++) {
+			vports[i].did = i;
+			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
+					iface, i);
+
+			start_vdpa(&vports[i]);
+		}
+
+		printf("enter \'q\' to quit\n");
+		while (scanf("%c", &ch)) {
+			if (ch == 'q')
+				break;
+			while (ch != '\n') {
+				if (scanf("%c", &ch))
+					printf("%c", ch);
+			}
+			printf("enter \'q\' to quit\n");
+		}
+		vdpa_sample_quit();
+	}
+
+	return 0;
+}
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
new file mode 100644
index 000000000..2e38a069f
--- /dev/null
+++ b/examples/vdpa/meson.build
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'vhost'
+allow_experimental_apis = true
+sources = files(
+	'main.c'
+)
\ No newline at end of file
-- 
2.17.1

Re: [PATCH v4 2/2] examples/vdpa: introduce a new sample for vDPA

[Wang, Xiao W]

Hi Xiaolong,

Thanks for the update, 2 small comments below.

> -----Original Message-----
> From: Ye, Xiaolong
> Sent: Monday, September 24, 2018 4:43 PM
> To: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Bie,
> Tiwei <tiwei.bie@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>
> Cc: Wang, Xiao W <xiao.w.wang@intel.com>; Rami Rosen
> <roszenrami@gmail.com>; Wang, Haiyue <haiyue.wang@intel.com>; Ye,
> Xiaolong <xiaolong.ye@intel.com>
> Subject: [PATCH v4 2/2] examples/vdpa: introduce a new sample for vDPA
> 
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>  MAINTAINERS                            |   2 +
>  doc/guides/rel_notes/release_18_11.rst |   8 +
>  doc/guides/sample_app_ug/index.rst     |   1 +
>  doc/guides/sample_app_ug/vdpa.rst      | 118 +++++++
>  examples/Makefile                      |   2 +-
>  examples/vdpa/Makefile                 |  32 ++
>  examples/vdpa/main.c                   | 466 +++++++++++++++++++++++++
>  examples/vdpa/meson.build              |  16 +
>  8 files changed, 644 insertions(+), 1 deletion(-)
>  create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>  create mode 100644 examples/vdpa/Makefile
>  create mode 100644 examples/vdpa/main.c
>  create mode 100644 examples/vdpa/meson.build
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
>  F: examples/vhost_scsi/
>  F: doc/guides/sample_app_ug/vhost_scsi.rst
>  F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
> 
>  Vhost PMD
>  M: Maxime Coquelin <maxime.coquelin@redhat.com>
> diff --git a/doc/guides/rel_notes/release_18_11.rst
> b/doc/guides/rel_notes/release_18_11.rst
> index bc9b74ec4..dd53a9ecf 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,14 @@ New Features
>    SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
>    vdev_netvsc, tap, and failsafe drivers combination.
> 
> +* **Add a new sample for vDPA**
> +
> +  The vdpa sample application creates vhost-user sockets by using the
> +  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +  virtio ring compatible devices to serve virtio driver directly to enable
> +  datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +  this application doesn't need to launch dedicated worker threads for vhost
> +  enqueue/dequeue operations.
> 
>  API Changes
>  -----------
> diff --git a/doc/guides/sample_app_ug/index.rst
> b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
>      vhost
>      vhost_scsi
>      vhost_crypto
> +    vdpa
>      netmap_compatibility
>      ip_pipeline
>      test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst
> b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..d05728a37
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,118 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps
> +-------------
> +
> +This section shows the steps of how to start VMs with vDPA vhost-user
> +backend and verify network connection & live migration.
> +
> +Build
> +~~~~~
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``vdpa`` sub-directory.
> +
> +Start the vdpa example
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
> +
> +where
> +
> +* --client means running vdpa app in client mode, in the client mode, QEMU
> needs
> +  to run as the server mode and take charge of socket file creation.
> +* --iface specifies the path prefix of the UNIX domain socket file, e.g.
> +  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
> +  (n starts from 0).
> +* --interactive means run the vdpa sample in interactive mode, currently 4
> +  internal cmds are supported:
> +
> +  1. help: show help message
> +  2. list: list all available vdpa devices
> +  3. create: create a new vdpa port with socket file and vdpa device address
> +  4. quit: unregister vhost driver and exit the application
> +
> +Take IFCVF driver for example:
> +
> +.. code-block:: console
> +
> +        ./vdpa --log-level=9 -c 0x6 -n 4 --socket-mem 1024,1024 \
> +                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
> +                -- --interactive

To demonstrate app doesn't need to launch dedicated worker threads for vhost enqueue/dequeue operations,
We can use "-c 0x2" to indicate that no need to allocate dedicated worker threads.

> +
> +.. note::
> +    We need to bind vfio-pci to VFs before running vdpa sample.
> +
> +    * modprobe vfio-pci
> +    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> +        vdpa> list
> +        device id       device address  queue num       supported features
> +        0               0000:06:00.3    1               0x5572362272
> +        1               0000:06:00.4    1               0x5572362272
> +
> +        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
> +        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 -cpu host -enable-kvm \
> +       <snip>
> +       -mem-prealloc \
> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> +       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> +    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.

[...]

> +
> +/* *** List all available vdpa devices *** */
> +struct cmd_list_result {
> +	cmdline_fixed_string_t action;
> +};
> +
> +static void cmd_list_vdpa_devices_parsed(
> +		__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	int did;
> +	uint32_t queue_num;
> +	uint64_t features;
> +	struct rte_vdpa_device *vdev;
> +	struct rte_pci_addr addr;
> +
> +	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported
> features\n");
> +	for (did = 0; did < dev_total; did++) {
> +		vdev = rte_vdpa_get_device(did);
> +		if (!vdev)
> +			continue;
> +		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa queue number "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		if (vdev->ops->get_features(did, &features) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa features "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		addr = vdev->addr.pci_addr;
> +		cmdline_printf(cl,
> +
> 	"%d\t\t"PCI_PRI_FMT"\t%"PRIu32"\t\t0x%"PRIu64"\n", did,
> +			addr.domain, addr.bus, addr.devid,
> +			addr.function, queue_num, features);

Use PRIx64 instead of PRIu64 for features.
You can add a blank space between "PRIx64" and the other section to make it more readable.
Refer to:
	lib/librte_vhost/vhost_user.c:                  "guest memory region %u, size: 0x%" PRIx64 "\n" 

BRs,
Xiao

Re: [PATCH v4 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

On 09/24, Wang, Xiao W wrote:
>Hi Xiaolong,
>
>Thanks for the update, 2 small comments below.
>
[snip]
>> +        ./vdpa --log-level=9 -c 0x6 -n 4 --socket-mem 1024,1024 \
>> +                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
>> +                -- --interactive
>
>To demonstrate app doesn't need to launch dedicated worker threads for vhost enqueue/dequeue operations,
>We can use "-c 0x2" to indicate that no need to allocate dedicated worker threads.
>

Got it, will do.

>> +
>> +.. note::
[snip]
>> 	"%d\t\t"PCI_PRI_FMT"\t%"PRIu32"\t\t0x%"PRIu64"\n", did,
>> +			addr.domain, addr.bus, addr.devid,
>> +			addr.function, queue_num, features);
>
>Use PRIx64 instead of PRIu64 for features.
>You can add a blank space between "PRIx64" and the other section to make it more readable.
>Refer to:
>	lib/librte_vhost/vhost_user.c:                  "guest memory region %u, size: 0x%" PRIx64 "\n" 

Got it, will do.

Thanks,
Xiaolong
>
>BRs,
>Xiao

[PATCH v5 0/2] introduce vdpa sample

[Xiaolong Ye]

Hi,

This patchset introduces vdpa sample to demonstrate the vDPA use case.

v5 changes:
* improve print format and correct from "PRIu64" to "PRIx64"
* use "-c 0x2" to better demonstrate app doesn't need to launch dedicated 
  worker threads for vhost enqueue/dequeue operations in vdpa.rst

v4 changes:
* add client mode support
* improve the format to list the vDPA device info and improve the vdpa.rst 
  accordingly
* remove some useless comments
* add introduction in 18.11 release note.


v3 changes:
* list cmd would show queue number and supported features of vdpa devices.
* address Xiao's review comments

v2 changes:

* fix a compilation error reported by Rosen
* improve create cmd in interactive mode and add two new cmds: list,
* quit
* add application documentation


Xiaolong Ye (2):
  vhost: introduce API to get vDPA device number
  examples/vdpa: introduce a new sample for vDPA

 MAINTAINERS                            |   2 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 115 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 458 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 lib/librte_vhost/rte_vdpa.h            |   3 +
 lib/librte_vhost/rte_vhost_version.map |   1 +
 lib/librte_vhost/vdpa.c                |   6 +
 10 files changed, 635 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

-- 
2.17.1

[PATCH v5 1/2] vhost: introduce API to get vDPA device number

[Xiaolong Ye]

It's used to get number of available registered vDPA devices.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 lib/librte_vhost/rte_vdpa.h            | 3 +++
 lib/librte_vhost/rte_vhost_version.map | 1 +
 lib/librte_vhost/vdpa.c                | 6 ++++++
 3 files changed, 10 insertions(+)

diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
index 90465ca26..b8223e337 100644
--- a/lib/librte_vhost/rte_vdpa.h
+++ b/lib/librte_vhost/rte_vdpa.h
@@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
 struct rte_vdpa_device * __rte_experimental
 rte_vdpa_get_device(int did);
 
+/* Get current available vdpa device number */
+int __rte_experimental
+rte_vdpa_get_device_num(void);
 #endif /* _RTE_VDPA_H_ */
diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
index da220dd02..ae39b6e21 100644
--- a/lib/librte_vhost/rte_vhost_version.map
+++ b/lib/librte_vhost/rte_vhost_version.map
@@ -67,6 +67,7 @@ EXPERIMENTAL {
 	rte_vdpa_unregister_device;
 	rte_vdpa_find_device_id;
 	rte_vdpa_get_device;
+	rte_vdpa_get_device_num;
 	rte_vhost_driver_attach_vdpa_device;
 	rte_vhost_driver_detach_vdpa_device;
 	rte_vhost_driver_get_vdpa_device_id;
diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
index c82fd4370..c2c5dff1d 100644
--- a/lib/librte_vhost/vdpa.c
+++ b/lib/librte_vhost/vdpa.c
@@ -113,3 +113,9 @@ rte_vdpa_get_device(int did)
 
 	return vdpa_devices[did];
 }
+
+int
+rte_vdpa_get_device_num(void)
+{
+	return vdpa_device_num;
+}
-- 
2.17.1

[PATCH v5 2/2] examples/vdpa: introduce a new sample for vDPA

[Xiaolong Ye]

The vdpa sample application creates vhost-user sockets by using the
vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
virtio ring compatible devices to serve virtio driver directly to enable
datapath acceleration. As vDPA driver can help to set up vhost datapath,
this application doesn't need to launch dedicated worker threads for vhost
enqueue/dequeue operations.

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 MAINTAINERS                            |   2 +
 doc/guides/rel_notes/release_18_11.rst |   8 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 118 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 466 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 8 files changed, 644 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

diff --git a/MAINTAINERS b/MAINTAINERS
index 5967c1dd3..5656f18e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
 F: examples/vhost_scsi/
 F: doc/guides/sample_app_ug/vhost_scsi.rst
 F: examples/vhost_crypto/
+F: examples/vdpa/
+F: doc/guides/sample_app_ug/vdpa.rst
 
 Vhost PMD
 M: Maxime Coquelin <maxime.coquelin@redhat.com>
diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index bc9b74ec4..dd53a9ecf 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -67,6 +67,14 @@ New Features
   SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
   vdev_netvsc, tap, and failsafe drivers combination.
 
+* **Add a new sample for vDPA**
+
+  The vdpa sample application creates vhost-user sockets by using the
+  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+  virtio ring compatible devices to serve virtio driver directly to enable
+  datapath acceleration. As vDPA driver can help to set up vhost datapath,
+  this application doesn't need to launch dedicated worker threads for vhost
+  enqueue/dequeue operations.
 
 API Changes
 -----------
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 5bedf4f6f..74b12af85 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -45,6 +45,7 @@ Sample Applications User Guides
     vhost
     vhost_scsi
     vhost_crypto
+    vdpa
     netmap_compatibility
     ip_pipeline
     test_pipeline
diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
new file mode 100644
index 000000000..a089393c0
--- /dev/null
+++ b/doc/guides/sample_app_ug/vdpa.rst
@@ -0,0 +1,118 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+Vdpa Sample Application
+=======================
+
+The vdpa sample application creates vhost-user sockets by using the
+vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+virtio ring compatible devices to serve virtio driver directly to enable
+datapath acceleration. As vDPA driver can help to set up vhost datapath,
+this application doesn't need to launch dedicated worker threads for vhost
+enqueue/dequeue operations.
+
+Testing steps
+-------------
+
+This section shows the steps of how to start VMs with vDPA vhost-user
+backend and verify network connection & live migration.
+
+Build
+~~~~~
+
+To compile the sample application see :doc:`compiling`.
+
+The application is located in the ``vdpa`` sub-directory.
+
+Start the vdpa example
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
+
+where
+
+* --client means running vdpa app in client mode, in the client mode, QEMU needs
+  to run as the server mode and take charge of socket file creation.
+* --iface specifies the path prefix of the UNIX domain socket file, e.g.
+  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
+  (n starts from 0).
+* --interactive means run the vdpa sample in interactive mode, currently 4
+  internal cmds are supported:
+
+  1. help: show help message
+  2. list: list all available vdpa devices
+  3. create: create a new vdpa port with socket file and vdpa device address
+  4. quit: unregister vhost driver and exit the application
+
+Take IFCVF driver for example:
+
+.. code-block:: console
+
+        ./vdpa --log-level=9 -c 0x2 -n 4 --socket-mem 1024,1024 \
+                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
+                -- --interactive
+
+.. note::
+    We need to bind vfio-pci to VFs before running vdpa sample.
+
+    * modprobe vfio-pci
+    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
+
+Then we can create 2 vdpa ports in interactive cmdline.
+
+.. code-block:: console
+
+        vdpa> list
+        device id       device address  queue num       supported features
+        0               0000:06:00.3    1               0x5572362272
+        1               0000:06:00.4    1               0x5572362272
+
+        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
+        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
+
+.. _vdpa_app_run_vm:
+
+Start the VMs
+~~~~~~~~~~~~~
+
+.. code-block:: console
+
+       qemu-system-x86_64 -cpu host -enable-kvm \
+       <snip>
+       -mem-prealloc \
+       -chardev socket,id=char0,path=<socket_file created in above steps> \
+       -netdev type=vhost-user,id=vdpa,chardev=char0 \
+       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \
+
+After the VMs launches, we can login the VMs and configure the ip, verify the
+network connection via ping or netperf.
+
+.. note::
+    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
+
+Live Migration
+~~~~~~~~~~~~~~
+vDPA supports cross-backend live migration, user can migrate SW vhost backend
+VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
+the source host with SW vhost VM and B is the destination host with vDPA.
+
+1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
+   in migration-listen mode:
+
+.. code-block:: console
+
+        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
+
+2. Start the migration (on source host):
+
+.. code-block:: console
+
+        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
+
+3. Check the status (on source host):
+
+.. code-block:: console
+
+        A: (qemu) info migrate
diff --git a/examples/Makefile b/examples/Makefile
index 481720cb6..356fcb1cd 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
-DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
+DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
 ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
 endif
diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
new file mode 100644
index 000000000..42672a2bc
--- /dev/null
+++ b/examples/vdpa/Makefile
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
+$(info This application can only operate in a linuxapp environment, \
+please change the definition of the RTE_TARGET environment variable)
+all:
+else
+
+# binary name
+APP = vdpa
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -D_GNU_SOURCE
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
new file mode 100644
index 000000000..7acda67c4
--- /dev/null
+++ b/examples/vdpa/main.c
@@ -0,0 +1,466 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <getopt.h>
+#include <signal.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_ethdev.h>
+#include <rte_malloc.h>
+#include <rte_vhost.h>
+#include <rte_vdpa.h>
+#include <rte_pci.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#define MAX_PATH_LEN 128
+#define MAX_VDPA_SAMPLE_PORTS 1024
+#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
+
+struct vdpa_port {
+	char ifname[MAX_PATH_LEN];
+	int did;
+	int vid;
+	uint64_t flags;
+};
+
+struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
+
+static char iface[MAX_PATH_LEN];
+static int dev_total;
+static int devcnt;
+static int interactive;
+static int client_mode;
+
+/* display usage */
+static void
+vdpa_usage(const char *prgname)
+{
+	printf("Usage: %s [EAL options] -- "
+				 "	--interactive|-i: run in interactive mode.\n"
+				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				 "	--client: register a vhost-user socket as client mode.\n",
+				 prgname);
+}
+
+static int
+parse_args(int argc, char **argv)
+{
+	static const char *short_option = "i";
+	static struct option long_option[] = {
+		{"iface", required_argument, NULL, 0},
+		{"interactive", no_argument, &interactive, 1},
+		{"client", no_argument, &client_mode, 1},
+		{NULL, 0, 0, 0},
+	};
+	int opt, idx;
+	char *prgname = argv[0];
+
+	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
+			!= EOF) {
+		switch (opt) {
+		case 'i':
+			printf("Interactive-mode selected\n");
+			interactive = 1;
+			break;
+		/* long options */
+		case 0:
+			if (strncmp(long_option[idx].name, "iface",
+						MAX_PATH_LEN) == 0) {
+				strncpy(iface, optarg, MAX_PATH_LEN);
+				printf("iface %s\n", iface);
+			}
+			if (!strcmp(long_option[idx].name, "interactive")) {
+				printf("Interactive-mode selected\n");
+				interactive = 1;
+			}
+			break;
+
+		default:
+			vdpa_usage(prgname);
+			return -1;
+		}
+	}
+
+	if (iface[0] == '\0' && interactive == 0) {
+		vdpa_usage(prgname);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+data_init(void)
+{
+	dev_total = rte_vdpa_get_device_num();
+	if (dev_total <= 0) {
+		printf("No available vdpa device found\n");
+		return -1;
+	}
+	devcnt = 0;
+	interactive = 0;
+	memset(iface, 0, sizeof(iface));
+	memset(vports, 0, sizeof(vports));
+	return 0;
+}
+
+static int
+new_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
+			printf("\nnew port %s, did: %d\n",
+					ifname, vports[i].did);
+			vports[i].vid = vid;
+			break;
+		}
+	}
+
+	if (i >= MAX_VDPA_SAMPLE_PORTS)
+		return -1;
+
+	return 0;
+}
+
+static void
+destroy_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strcmp(ifname, vports[i].ifname) == 0) {
+			printf("\ndestroy port %s, did: %d\n",
+					ifname, vports[i].did);
+			break;
+		}
+	}
+}
+
+static const struct vhost_device_ops vdpa_sample_devops = {
+	.new_device = new_device,
+	.destroy_device = destroy_device,
+};
+
+static int
+start_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+	int did = vport->did;
+
+	if (client_mode)
+		vport->flags |= RTE_VHOST_USER_CLIENT;
+
+	if (access(socket_path, F_OK) != -1 && !client_mode) {
+		RTE_LOG(ERR, VDPA,
+			"%s exists, please remove it or specify another file and try again.\n",
+			socket_path);
+		return -1;
+	}
+	ret = rte_vhost_driver_register(socket_path, vport->flags);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_callback_register(socket_path,
+			&vdpa_sample_devops);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver ops failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"attach vdpa device failed: %s\n",
+			socket_path);
+
+	if (rte_vhost_driver_start(socket_path) < 0)
+		rte_exit(EXIT_FAILURE,
+			"start vhost driver failed: %s\n",
+			socket_path);
+	return 0;
+}
+
+static void
+close_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+
+	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"detach vdpa device failed: %s\n",
+				socket_path);
+
+	ret = rte_vhost_driver_unregister(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"Fail to unregister vhost driver for %s.\n",
+				socket_path);
+}
+
+static void
+vdpa_sample_quit(void)
+{
+	int i;
+	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
+		if (vports[i].ifname[0] != '\0')
+			close_vdpa(&vports[i]);
+	}
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\nSignal %d received, preparing to exit...\n", signum);
+		vdpa_sample_quit();
+		exit(0);
+	}
+}
+
+/* interactive cmds */
+
+/* *** Help command with introduction. *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	cmdline_printf(
+		cl,
+		"\n"
+		"The following commands are currently available:\n\n"
+		"Control:\n"
+		"    help                                      : Show interactive instructions.\n"
+		"    list                                      : list all available vdpa devices.\n"
+		"    create <socket file> <vdev addr>          : create a new vdpa port.\n"
+		"    quit                                      : exit vdpa sample app.\n"
+	);
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show help",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/* *** List all available vdpa devices *** */
+struct cmd_list_result {
+	cmdline_fixed_string_t action;
+};
+
+static void cmd_list_vdpa_devices_parsed(
+		__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	uint32_t queue_num;
+	uint64_t features;
+	struct rte_vdpa_device *vdev;
+	struct rte_pci_addr addr;
+
+	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
+	for (did = 0; did < dev_total; did++) {
+		vdev = rte_vdpa_get_device(did);
+		if (!vdev)
+			continue;
+		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa queue number "
+				"for device id %d.\n", did);
+			continue;
+		}
+		if (vdev->ops->get_features(did, &features) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa features "
+				"for device id %d.\n", did);
+			continue;
+		}
+		addr = vdev->addr.pci_addr;
+		cmdline_printf(cl,
+			"%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIu64 "\n", did,
+			addr.domain, addr.bus, addr.devid,
+			addr.function, queue_num, features);
+	}
+}
+
+cmdline_parse_token_string_t cmd_action_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
+
+cmdline_parse_inst_t cmd_list_vdpa_devices = {
+	.f = cmd_list_vdpa_devices_parsed,
+	.data = NULL,
+	.help_str = "list all available vdpa devices",
+	.tokens = {
+		(void *)&cmd_action_list,
+		NULL,
+	},
+};
+
+/* *** Create new vdpa port *** */
+struct cmd_create_result {
+	cmdline_fixed_string_t action;
+	cmdline_fixed_string_t socket_path;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_create_vdpa_port_parsed(void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	struct cmd_create_result *res = parsed_result;
+	struct rte_vdpa_dev_addr addr;
+
+	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
+	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
+		cmdline_printf(cl, "Unable to parse the given bdf.\n");
+		return;
+	}
+	addr.type = PCI_ADDR;
+	did = rte_vdpa_find_device_id(&addr);
+	if (did < 0) {
+		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
+		return;
+	}
+
+	vports[devcnt].did = did;
+
+	if (start_vdpa(&vports[devcnt]) == 0)
+		devcnt++;
+}
+
+cmdline_parse_token_string_t cmd_action_create =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
+cmdline_parse_token_string_t cmd_socket_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
+cmdline_parse_token_string_t cmd_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_create_vdpa_port = {
+	.f = cmd_create_vdpa_port_parsed,
+	.data = NULL,
+	.help_str = "create a new vdpa port",
+	.tokens = {
+		(void *)&cmd_action_create,
+		(void *)&cmd_socket_path,
+		(void *)&cmd_bdf,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	vdpa_sample_quit();
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "quit: exit application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
+	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	NULL,
+};
+
+int
+main(int argc, char *argv[])
+{
+	char ch;
+	int i;
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "eal init failed\n");
+	argc -= ret;
+	argv += ret;
+
+	data_init();
+
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	ret = parse_args(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "invalid argument\n");
+
+	if (interactive == 1) {
+		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
+		if (cl == NULL)
+			rte_panic("Cannot create cmdline instance\n");
+		cmdline_interact(cl);
+		cmdline_stdin_exit(cl);
+	} else {
+		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
+				i++) {
+			vports[i].did = i;
+			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
+					iface, i);
+
+			start_vdpa(&vports[i]);
+		}
+
+		printf("enter \'q\' to quit\n");
+		while (scanf("%c", &ch)) {
+			if (ch == 'q')
+				break;
+			while (ch != '\n') {
+				if (scanf("%c", &ch))
+					printf("%c", ch);
+			}
+			printf("enter \'q\' to quit\n");
+		}
+		vdpa_sample_quit();
+	}
+
+	return 0;
+}
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
new file mode 100644
index 000000000..2e38a069f
--- /dev/null
+++ b/examples/vdpa/meson.build
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'vhost'
+allow_experimental_apis = true
+sources = files(
+	'main.c'
+)
\ No newline at end of file
-- 
2.17.1

Re: [PATCH v5 2/2] examples/vdpa: introduce a new sample for vDPA

[Wang, Xiao W]

Hi,

> -----Original Message-----
> From: Ye, Xiaolong
> Sent: Tuesday, September 25, 2018 8:07 PM
> To: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Bie,
> Tiwei <tiwei.bie@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>
> Cc: Wang, Xiao W <xiao.w.wang@intel.com>; Rami Rosen
> <roszenrami@gmail.com>; Wang, Haiyue <haiyue.wang@intel.com>; Ye,
> Xiaolong <xiaolong.ye@intel.com>
> Subject: [PATCH v5 2/2] examples/vdpa: introduce a new sample for vDPA
> 
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>  MAINTAINERS                            |   2 +
>  doc/guides/rel_notes/release_18_11.rst |   8 +
>  doc/guides/sample_app_ug/index.rst     |   1 +
>  doc/guides/sample_app_ug/vdpa.rst      | 118 +++++++
>  examples/Makefile                      |   2 +-
>  examples/vdpa/Makefile                 |  32 ++
>  examples/vdpa/main.c                   | 466 +++++++++++++++++++++++++
>  examples/vdpa/meson.build              |  16 +
>  8 files changed, 644 insertions(+), 1 deletion(-)
>  create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>  create mode 100644 examples/vdpa/Makefile
>  create mode 100644 examples/vdpa/main.c
>  create mode 100644 examples/vdpa/meson.build
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
>  F: examples/vhost_scsi/
>  F: doc/guides/sample_app_ug/vhost_scsi.rst
>  F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
> 
>  Vhost PMD
>  M: Maxime Coquelin <maxime.coquelin@redhat.com>
> diff --git a/doc/guides/rel_notes/release_18_11.rst
> b/doc/guides/rel_notes/release_18_11.rst
> index bc9b74ec4..dd53a9ecf 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,14 @@ New Features
>    SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
>    vdev_netvsc, tap, and failsafe drivers combination.
> 
> +* **Add a new sample for vDPA**
> +
> +  The vdpa sample application creates vhost-user sockets by using the
> +  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +  virtio ring compatible devices to serve virtio driver directly to enable
> +  datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +  this application doesn't need to launch dedicated worker threads for vhost
> +  enqueue/dequeue operations.
> 
>  API Changes
>  -----------
> diff --git a/doc/guides/sample_app_ug/index.rst
> b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
>      vhost
>      vhost_scsi
>      vhost_crypto
> +    vdpa
>      netmap_compatibility
>      ip_pipeline
>      test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst
> b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..a089393c0
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,118 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps
> +-------------
> +
> +This section shows the steps of how to start VMs with vDPA vhost-user
> +backend and verify network connection & live migration.
> +
> +Build
> +~~~~~
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``vdpa`` sub-directory.
> +
> +Start the vdpa example
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface
> SOCKET_PATH]
> +
> +where
> +
> +* --client means running vdpa app in client mode, in the client mode, QEMU
> needs
> +  to run as the server mode and take charge of socket file creation.
> +* --iface specifies the path prefix of the UNIX domain socket file, e.g.
> +  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
> +  (n starts from 0).
> +* --interactive means run the vdpa sample in interactive mode, currently 4
> +  internal cmds are supported:
> +
> +  1. help: show help message
> +  2. list: list all available vdpa devices
> +  3. create: create a new vdpa port with socket file and vdpa device address
> +  4. quit: unregister vhost driver and exit the application
> +
> +Take IFCVF driver for example:
> +
> +.. code-block:: console
> +
> +        ./vdpa --log-level=9 -c 0x2 -n 4 --socket-mem 1024,1024 \
> +                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
> +                -- --interactive

DPDK has log level 1-8, setting the log-level=9 looks strange.
Here you can add description about the whitelisted device, since they are
virtio ring compatible devices.

> +
> +.. note::
> +    We need to bind vfio-pci to VFs before running vdpa sample.
> +
> +    * modprobe vfio-pci
> +    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> +        vdpa> list
> +        device id       device address  queue num       supported features
> +        0               0000:06:00.3    1               0x5572362272
> +        1               0000:06:00.4    1               0x5572362272
> +
> +        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
> +        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 -cpu host -enable-kvm \
> +       <snip>
> +       -mem-prealloc \
> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> +       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee \

In order to leverage EPT, add parameter "page-per-vq=on" for the virtio-net-pci deivce.

> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> +    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
> +
> +Live Migration
> +~~~~~~~~~~~~~~
> +vDPA supports cross-backend live migration, user can migrate SW vhost
> backend
> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume
> A is
> +the source host with SW vhost VM and B is the destination host with vDPA.
> +
> +1. Start vdpa sample and launch a VM with exact same parameters as the VM
> on A,
> +   in migration-listen mode:
> +
> +.. code-block:: console
> +
> +        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
> +
> +2. Start the migration (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
> +
> +3. Check the status (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) info migrate
> diff --git a/examples/Makefile b/examples/Makefile
> index 481720cb6..356fcb1cd 100644
> --- a/examples/Makefile
> +++ b/examples/Makefile
> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
>  DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
>  endif
>  DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
>  ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
>  DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
>  endif
> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
> new file mode 100644
> index 000000000..42672a2bc
> --- /dev/null
> +++ b/examples/vdpa/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +ifeq ($(RTE_SDK),)
> +$(error "Please define RTE_SDK environment variable")
> +endif
> +
> +# Default target, can be overridden by command line or environment
> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
> +$(info This application can only operate in a linuxapp environment, \
> +please change the definition of the RTE_TARGET environment variable)
> +all:
> +else
> +
> +# binary name
> +APP = vdpa
> +
> +# all source are stored in SRCS-y
> +SRCS-y := main.c
> +
> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -D_GNU_SOURCE
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +
> +include $(RTE_SDK)/mk/rte.extapp.mk
> +
> +endif
> diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
> new file mode 100644
> index 000000000..7acda67c4
> --- /dev/null
> +++ b/examples/vdpa/main.c
> @@ -0,0 +1,466 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#include <getopt.h>
> +#include <signal.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#include <rte_ethdev.h>
> +#include <rte_malloc.h>
> +#include <rte_vhost.h>
> +#include <rte_vdpa.h>
> +#include <rte_pci.h>
> +
> +#include <cmdline_parse.h>
> +#include <cmdline_socket.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline.h>
> +
> +#define MAX_PATH_LEN 128
> +#define MAX_VDPA_SAMPLE_PORTS 1024
> +#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
> +

[...]

> +};
> +
> +/* *** List all available vdpa devices *** */
> +struct cmd_list_result {
> +	cmdline_fixed_string_t action;
> +};
> +
> +static void cmd_list_vdpa_devices_parsed(
> +		__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	int did;
> +	uint32_t queue_num;
> +	uint64_t features;
> +	struct rte_vdpa_device *vdev;
> +	struct rte_pci_addr addr;
> +
> +	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported
> features\n");
> +	for (did = 0; did < dev_total; did++) {
> +		vdev = rte_vdpa_get_device(did);
> +		if (!vdev)
> +			continue;
> +		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa queue number "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		if (vdev->ops->get_features(did, &features) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa features "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		addr = vdev->addr.pci_addr;
> +		cmdline_printf(cl,
> +			"%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIu64
> "\n", did,

s/PRIu64/PRIx64

> +			addr.domain, addr.bus, addr.devid,
> +			addr.function, queue_num, features);
> +	}
> +}

BRs,
Xiao

[PATCH v6 0/2] introduce vdpa sample

[Xiaolong Ye]

Hi,

This patchset introduces vdpa sample to demonstrate the vDPA use case.

v6 changes:
* improve the document according to Xiao's comments
* fix a typo, PRIu64 -> PRIx64

v5 changes:
* improve print format and correct from "PRIu64" to "PRIx64"
* use "-c 0x2" to better demonstrate app doesn't need to launch dedicated 
  worker threads for vhost enqueue/dequeue operations in vdpa.rst

v4 changes:
* add client mode support
* improve the format to list the vDPA device info and improve the vdpa.rst 
  accordingly
* remove some useless comments
* add introduction in 18.11 release note.


v3 changes:
* list cmd would show queue number and supported features of vdpa devices.
* address Xiao's review comments

v2 changes:

* fix a compilation error reported by Rosen
* improve create cmd in interactive mode and add two new cmds: list, quit
* add application documentation


Xiaolong Ye (2):
  vhost: introduce API to get vDPA device number
  examples/vdpa: introduce a new sample for vDPA

 MAINTAINERS                            |   2 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 115 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 458 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 lib/librte_vhost/rte_vdpa.h            |   3 +
 lib/librte_vhost/rte_vhost_version.map |   1 +
 lib/librte_vhost/vdpa.c                |   6 +
 10 files changed, 635 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

-- 
2.17.1

Re: [PATCH v6 0/2] introduce vdpa sample

[Wang, Xiao W]

Hi,

> -----Original Message-----
> From: Ye, Xiaolong
> Sent: Wednesday, September 26, 2018 5:07 PM
> To: dev@dpdk.org; Maxime Coquelin <maxime.coquelin@redhat.com>; Bie,
> Tiwei <tiwei.bie@intel.com>; Wang, Zhihong <zhihong.wang@intel.com>
> Cc: Wang, Xiao W <xiao.w.wang@intel.com>; Rami Rosen
> <roszenrami@gmail.com>; Wang, Haiyue <haiyue.wang@intel.com>; Ye,
> Xiaolong <xiaolong.ye@intel.com>
> Subject: [PATCH v6 0/2] introduce vdpa sample
> 
> Hi,
> 
> This patchset introduces vdpa sample to demonstrate the vDPA use case.
> 
> v6 changes:
> * improve the document according to Xiao's comments
> * fix a typo, PRIu64 -> PRIx64
> 
> v5 changes:
> * improve print format and correct from "PRIu64" to "PRIx64"
> * use "-c 0x2" to better demonstrate app doesn't need to launch dedicated
>   worker threads for vhost enqueue/dequeue operations in vdpa.rst
> 
> v4 changes:
> * add client mode support
> * improve the format to list the vDPA device info and improve the vdpa.rst
>   accordingly
> * remove some useless comments
> * add introduction in 18.11 release note.
> 
> 
> v3 changes:
> * list cmd would show queue number and supported features of vdpa devices.
> * address Xiao's review comments
> 
> v2 changes:
> 
> * fix a compilation error reported by Rosen
> * improve create cmd in interactive mode and add two new cmds: list, quit
> * add application documentation
> 
> 
> Xiaolong Ye (2):
>   vhost: introduce API to get vDPA device number
>   examples/vdpa: introduce a new sample for vDPA
> 
>  MAINTAINERS                            |   2 +
>  doc/guides/sample_app_ug/index.rst     |   1 +
>  doc/guides/sample_app_ug/vdpa.rst      | 115 +++++++
>  examples/Makefile                      |   2 +-
>  examples/vdpa/Makefile                 |  32 ++
>  examples/vdpa/main.c                   | 458 +++++++++++++++++++++++++
>  examples/vdpa/meson.build              |  16 +
>  lib/librte_vhost/rte_vdpa.h            |   3 +
>  lib/librte_vhost/rte_vhost_version.map |   1 +
>  lib/librte_vhost/vdpa.c                |   6 +
>  10 files changed, 635 insertions(+), 1 deletion(-)
>  create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>  create mode 100644 examples/vdpa/Makefile
>  create mode 100644 examples/vdpa/main.c
>  create mode 100644 examples/vdpa/meson.build
> 
> --
> 2.17.1

Series Acked-by: Xiao Wang <xiao.w.wang@intel.com>

BRs,
Xiao

[PATCH v6 1/2] vhost: introduce API to get vDPA device number

[Xiaolong Ye]

It's used to get number of available registered vDPA devices.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 lib/librte_vhost/rte_vdpa.h            | 3 +++
 lib/librte_vhost/rte_vhost_version.map | 1 +
 lib/librte_vhost/vdpa.c                | 6 ++++++
 3 files changed, 10 insertions(+)

diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
index 90465ca26..b8223e337 100644
--- a/lib/librte_vhost/rte_vdpa.h
+++ b/lib/librte_vhost/rte_vdpa.h
@@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
 struct rte_vdpa_device * __rte_experimental
 rte_vdpa_get_device(int did);
 
+/* Get current available vdpa device number */
+int __rte_experimental
+rte_vdpa_get_device_num(void);
 #endif /* _RTE_VDPA_H_ */
diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
index da220dd02..ae39b6e21 100644
--- a/lib/librte_vhost/rte_vhost_version.map
+++ b/lib/librte_vhost/rte_vhost_version.map
@@ -67,6 +67,7 @@ EXPERIMENTAL {
 	rte_vdpa_unregister_device;
 	rte_vdpa_find_device_id;
 	rte_vdpa_get_device;
+	rte_vdpa_get_device_num;
 	rte_vhost_driver_attach_vdpa_device;
 	rte_vhost_driver_detach_vdpa_device;
 	rte_vhost_driver_get_vdpa_device_id;
diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
index c82fd4370..c2c5dff1d 100644
--- a/lib/librte_vhost/vdpa.c
+++ b/lib/librte_vhost/vdpa.c
@@ -113,3 +113,9 @@ rte_vdpa_get_device(int did)
 
 	return vdpa_devices[did];
 }
+
+int
+rte_vdpa_get_device_num(void)
+{
+	return vdpa_device_num;
+}
-- 
2.17.1

[PATCH v6 2/2] examples/vdpa: introduce a new sample for vDPA

[Xiaolong Ye]

The vdpa sample application creates vhost-user sockets by using the
vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
virtio ring compatible devices to serve virtio driver directly to enable
datapath acceleration. As vDPA driver can help to set up vhost datapath,
this application doesn't need to launch dedicated worker threads for vhost
enqueue/dequeue operations.

Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 MAINTAINERS                            |   2 +
 doc/guides/rel_notes/release_18_11.rst |   8 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 466 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 8 files changed, 646 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

diff --git a/MAINTAINERS b/MAINTAINERS
index 5967c1dd3..5656f18e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
 F: examples/vhost_scsi/
 F: doc/guides/sample_app_ug/vhost_scsi.rst
 F: examples/vhost_crypto/
+F: examples/vdpa/
+F: doc/guides/sample_app_ug/vdpa.rst
 
 Vhost PMD
 M: Maxime Coquelin <maxime.coquelin@redhat.com>
diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index bc9b74ec4..dd53a9ecf 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -67,6 +67,14 @@ New Features
   SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
   vdev_netvsc, tap, and failsafe drivers combination.
 
+* **Add a new sample for vDPA**
+
+  The vdpa sample application creates vhost-user sockets by using the
+  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+  virtio ring compatible devices to serve virtio driver directly to enable
+  datapath acceleration. As vDPA driver can help to set up vhost datapath,
+  this application doesn't need to launch dedicated worker threads for vhost
+  enqueue/dequeue operations.
 
 API Changes
 -----------
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 5bedf4f6f..74b12af85 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -45,6 +45,7 @@ Sample Applications User Guides
     vhost
     vhost_scsi
     vhost_crypto
+    vdpa
     netmap_compatibility
     ip_pipeline
     test_pipeline
diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
new file mode 100644
index 000000000..745f196ca
--- /dev/null
+++ b/doc/guides/sample_app_ug/vdpa.rst
@@ -0,0 +1,120 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+Vdpa Sample Application
+=======================
+
+The vdpa sample application creates vhost-user sockets by using the
+vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+virtio ring compatible devices to serve virtio driver directly to enable
+datapath acceleration. As vDPA driver can help to set up vhost datapath,
+this application doesn't need to launch dedicated worker threads for vhost
+enqueue/dequeue operations.
+
+Testing steps
+-------------
+
+This section shows the steps of how to start VMs with vDPA vhost-user
+backend and verify network connection & live migration.
+
+Build
+~~~~~
+
+To compile the sample application see :doc:`compiling`.
+
+The application is located in the ``vdpa`` sub-directory.
+
+Start the vdpa example
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
+
+where
+
+* --client means running vdpa app in client mode, in the client mode, QEMU needs
+  to run as the server mode and take charge of socket file creation.
+* --iface specifies the path prefix of the UNIX domain socket file, e.g.
+  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
+  (n starts from 0).
+* --interactive means run the vdpa sample in interactive mode, currently 4
+  internal cmds are supported:
+
+  1. help: show help message
+  2. list: list all available vdpa devices
+  3. create: create a new vdpa port with socket file and vdpa device address
+  4. quit: unregister vhost driver and exit the application
+
+Take IFCVF driver for example:
+
+.. code-block:: console
+
+        ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \
+                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
+                -- --interactive
+
+.. note::
+    Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices,
+    and we need to bind vfio-pci to them before running vdpa sample.
+
+    * modprobe vfio-pci
+    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
+
+Then we can create 2 vdpa ports in interactive cmdline.
+
+.. code-block:: console
+
+        vdpa> list
+        device id       device address  queue num       supported features
+        0               0000:06:00.3    1               0x14c238020
+        1               0000:06:00.4    1               0x14c238020
+        2               0000:06:00.5    1               0x14c238020
+
+        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
+        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
+
+.. _vdpa_app_run_vm:
+
+Start the VMs
+~~~~~~~~~~~~~
+
+.. code-block:: console
+
+       qemu-system-x86_64 -cpu host -enable-kvm \
+       <snip>
+       -mem-prealloc \
+       -chardev socket,id=char0,path=<socket_file created in above steps> \
+       -netdev type=vhost-user,id=vdpa,chardev=char0 \
+       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
+
+After the VMs launches, we can login the VMs and configure the ip, verify the
+network connection via ping or netperf.
+
+.. note::
+    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
+
+Live Migration
+~~~~~~~~~~~~~~
+vDPA supports cross-backend live migration, user can migrate SW vhost backend
+VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
+the source host with SW vhost VM and B is the destination host with vDPA.
+
+1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
+   in migration-listen mode:
+
+.. code-block:: console
+
+        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
+
+2. Start the migration (on source host):
+
+.. code-block:: console
+
+        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
+
+3. Check the status (on source host):
+
+.. code-block:: console
+
+        A: (qemu) info migrate
diff --git a/examples/Makefile b/examples/Makefile
index 481720cb6..356fcb1cd 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
-DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
+DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
 ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
 endif
diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
new file mode 100644
index 000000000..42672a2bc
--- /dev/null
+++ b/examples/vdpa/Makefile
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
+$(info This application can only operate in a linuxapp environment, \
+please change the definition of the RTE_TARGET environment variable)
+all:
+else
+
+# binary name
+APP = vdpa
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -D_GNU_SOURCE
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
new file mode 100644
index 000000000..898286e30
--- /dev/null
+++ b/examples/vdpa/main.c
@@ -0,0 +1,466 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <getopt.h>
+#include <signal.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_ethdev.h>
+#include <rte_malloc.h>
+#include <rte_vhost.h>
+#include <rte_vdpa.h>
+#include <rte_pci.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#define MAX_PATH_LEN 128
+#define MAX_VDPA_SAMPLE_PORTS 1024
+#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
+
+struct vdpa_port {
+	char ifname[MAX_PATH_LEN];
+	int did;
+	int vid;
+	uint64_t flags;
+};
+
+struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
+
+static char iface[MAX_PATH_LEN];
+static int dev_total;
+static int devcnt;
+static int interactive;
+static int client_mode;
+
+/* display usage */
+static void
+vdpa_usage(const char *prgname)
+{
+	printf("Usage: %s [EAL options] -- "
+				 "	--interactive|-i: run in interactive mode.\n"
+				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				 "	--client: register a vhost-user socket as client mode.\n",
+				 prgname);
+}
+
+static int
+parse_args(int argc, char **argv)
+{
+	static const char *short_option = "i";
+	static struct option long_option[] = {
+		{"iface", required_argument, NULL, 0},
+		{"interactive", no_argument, &interactive, 1},
+		{"client", no_argument, &client_mode, 1},
+		{NULL, 0, 0, 0},
+	};
+	int opt, idx;
+	char *prgname = argv[0];
+
+	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
+			!= EOF) {
+		switch (opt) {
+		case 'i':
+			printf("Interactive-mode selected\n");
+			interactive = 1;
+			break;
+		/* long options */
+		case 0:
+			if (strncmp(long_option[idx].name, "iface",
+						MAX_PATH_LEN) == 0) {
+				strncpy(iface, optarg, MAX_PATH_LEN);
+				printf("iface %s\n", iface);
+			}
+			if (!strcmp(long_option[idx].name, "interactive")) {
+				printf("Interactive-mode selected\n");
+				interactive = 1;
+			}
+			break;
+
+		default:
+			vdpa_usage(prgname);
+			return -1;
+		}
+	}
+
+	if (iface[0] == '\0' && interactive == 0) {
+		vdpa_usage(prgname);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+data_init(void)
+{
+	dev_total = rte_vdpa_get_device_num();
+	if (dev_total <= 0) {
+		printf("No available vdpa device found\n");
+		return -1;
+	}
+	devcnt = 0;
+	interactive = 0;
+	memset(iface, 0, sizeof(iface));
+	memset(vports, 0, sizeof(vports));
+	return 0;
+}
+
+static int
+new_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
+			printf("\nnew port %s, did: %d\n",
+					ifname, vports[i].did);
+			vports[i].vid = vid;
+			break;
+		}
+	}
+
+	if (i >= MAX_VDPA_SAMPLE_PORTS)
+		return -1;
+
+	return 0;
+}
+
+static void
+destroy_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strcmp(ifname, vports[i].ifname) == 0) {
+			printf("\ndestroy port %s, did: %d\n",
+					ifname, vports[i].did);
+			break;
+		}
+	}
+}
+
+static const struct vhost_device_ops vdpa_sample_devops = {
+	.new_device = new_device,
+	.destroy_device = destroy_device,
+};
+
+static int
+start_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+	int did = vport->did;
+
+	if (client_mode)
+		vport->flags |= RTE_VHOST_USER_CLIENT;
+
+	if (access(socket_path, F_OK) != -1 && !client_mode) {
+		RTE_LOG(ERR, VDPA,
+			"%s exists, please remove it or specify another file and try again.\n",
+			socket_path);
+		return -1;
+	}
+	ret = rte_vhost_driver_register(socket_path, vport->flags);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_callback_register(socket_path,
+			&vdpa_sample_devops);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver ops failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"attach vdpa device failed: %s\n",
+			socket_path);
+
+	if (rte_vhost_driver_start(socket_path) < 0)
+		rte_exit(EXIT_FAILURE,
+			"start vhost driver failed: %s\n",
+			socket_path);
+	return 0;
+}
+
+static void
+close_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+
+	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"detach vdpa device failed: %s\n",
+				socket_path);
+
+	ret = rte_vhost_driver_unregister(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"Fail to unregister vhost driver for %s.\n",
+				socket_path);
+}
+
+static void
+vdpa_sample_quit(void)
+{
+	int i;
+	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
+		if (vports[i].ifname[0] != '\0')
+			close_vdpa(&vports[i]);
+	}
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\nSignal %d received, preparing to exit...\n", signum);
+		vdpa_sample_quit();
+		exit(0);
+	}
+}
+
+/* interactive cmds */
+
+/* *** Help command with introduction. *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	cmdline_printf(
+		cl,
+		"\n"
+		"The following commands are currently available:\n\n"
+		"Control:\n"
+		"    help                                      : Show interactive instructions.\n"
+		"    list                                      : list all available vdpa devices.\n"
+		"    create <socket file> <vdev addr>          : create a new vdpa port.\n"
+		"    quit                                      : exit vdpa sample app.\n"
+	);
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show help",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/* *** List all available vdpa devices *** */
+struct cmd_list_result {
+	cmdline_fixed_string_t action;
+};
+
+static void cmd_list_vdpa_devices_parsed(
+		__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	uint32_t queue_num;
+	uint64_t features;
+	struct rte_vdpa_device *vdev;
+	struct rte_pci_addr addr;
+
+	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
+	for (did = 0; did < dev_total; did++) {
+		vdev = rte_vdpa_get_device(did);
+		if (!vdev)
+			continue;
+		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa queue number "
+				"for device id %d.\n", did);
+			continue;
+		}
+		if (vdev->ops->get_features(did, &features) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa features "
+				"for device id %d.\n", did);
+			continue;
+		}
+		addr = vdev->addr.pci_addr;
+		cmdline_printf(cl,
+			"%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIx64 "\n",
+			did, addr.domain, addr.bus, addr.devid,
+			addr.function, queue_num, features);
+	}
+}
+
+cmdline_parse_token_string_t cmd_action_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
+
+cmdline_parse_inst_t cmd_list_vdpa_devices = {
+	.f = cmd_list_vdpa_devices_parsed,
+	.data = NULL,
+	.help_str = "list all available vdpa devices",
+	.tokens = {
+		(void *)&cmd_action_list,
+		NULL,
+	},
+};
+
+/* *** Create new vdpa port *** */
+struct cmd_create_result {
+	cmdline_fixed_string_t action;
+	cmdline_fixed_string_t socket_path;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_create_vdpa_port_parsed(void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	struct cmd_create_result *res = parsed_result;
+	struct rte_vdpa_dev_addr addr;
+
+	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
+	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
+		cmdline_printf(cl, "Unable to parse the given bdf.\n");
+		return;
+	}
+	addr.type = PCI_ADDR;
+	did = rte_vdpa_find_device_id(&addr);
+	if (did < 0) {
+		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
+		return;
+	}
+
+	vports[devcnt].did = did;
+
+	if (start_vdpa(&vports[devcnt]) == 0)
+		devcnt++;
+}
+
+cmdline_parse_token_string_t cmd_action_create =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
+cmdline_parse_token_string_t cmd_socket_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
+cmdline_parse_token_string_t cmd_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_create_vdpa_port = {
+	.f = cmd_create_vdpa_port_parsed,
+	.data = NULL,
+	.help_str = "create a new vdpa port",
+	.tokens = {
+		(void *)&cmd_action_create,
+		(void *)&cmd_socket_path,
+		(void *)&cmd_bdf,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	vdpa_sample_quit();
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "quit: exit application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
+	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	NULL,
+};
+
+int
+main(int argc, char *argv[])
+{
+	char ch;
+	int i;
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "eal init failed\n");
+	argc -= ret;
+	argv += ret;
+
+	data_init();
+
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	ret = parse_args(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "invalid argument\n");
+
+	if (interactive == 1) {
+		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
+		if (cl == NULL)
+			rte_panic("Cannot create cmdline instance\n");
+		cmdline_interact(cl);
+		cmdline_stdin_exit(cl);
+	} else {
+		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
+				i++) {
+			vports[i].did = i;
+			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
+					iface, i);
+
+			start_vdpa(&vports[i]);
+		}
+
+		printf("enter \'q\' to quit\n");
+		while (scanf("%c", &ch)) {
+			if (ch == 'q')
+				break;
+			while (ch != '\n') {
+				if (scanf("%c", &ch))
+					printf("%c", ch);
+			}
+			printf("enter \'q\' to quit\n");
+		}
+		vdpa_sample_quit();
+	}
+
+	return 0;
+}
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
new file mode 100644
index 000000000..2e38a069f
--- /dev/null
+++ b/examples/vdpa/meson.build
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'vhost'
+allow_experimental_apis = true
+sources = files(
+	'main.c'
+)
\ No newline at end of file
-- 
2.17.1

Re: [PATCH v6 2/2] examples/vdpa: introduce a new sample for vDPA

[Maxime Coquelin]

On 09/26/2018 11:06 AM, Xiaolong Ye wrote:
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Signed-off-by: Xiao Wang <xiao.w.wang@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>   MAINTAINERS                            |   2 +
>   doc/guides/rel_notes/release_18_11.rst |   8 +
>   doc/guides/sample_app_ug/index.rst     |   1 +
>   doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
>   examples/Makefile                      |   2 +-
>   examples/vdpa/Makefile                 |  32 ++
>   examples/vdpa/main.c                   | 466 +++++++++++++++++++++++++
>   examples/vdpa/meson.build              |  16 +
>   8 files changed, 646 insertions(+), 1 deletion(-)
>   create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>   create mode 100644 examples/vdpa/Makefile
>   create mode 100644 examples/vdpa/main.c
>   create mode 100644 examples/vdpa/meson.build
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
>   F: examples/vhost_scsi/
>   F: doc/guides/sample_app_ug/vhost_scsi.rst
>   F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
>   
>   Vhost PMD
>   M: Maxime Coquelin <maxime.coquelin@redhat.com>
> diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
> index bc9b74ec4..dd53a9ecf 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,14 @@ New Features
>     SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
>     vdev_netvsc, tap, and failsafe drivers combination.
>   
> +* **Add a new sample for vDPA**
> +
> +  The vdpa sample application creates vhost-user sockets by using the
> +  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +  virtio ring compatible devices to serve virtio driver directly to enable
> +  datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +  this application doesn't need to launch dedicated worker threads for vhost
> +  enqueue/dequeue operations.
>   
>   API Changes
>   -----------
> diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
>       vhost
>       vhost_scsi
>       vhost_crypto
> +    vdpa
>       netmap_compatibility
>       ip_pipeline
>       test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..745f196ca
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,120 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps
> +-------------
> +
> +This section shows the steps of how to start VMs with vDPA vhost-user
> +backend and verify network connection & live migration.
> +
> +Build
> +~~~~~
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``vdpa`` sub-directory.
> +
> +Start the vdpa example
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
> +
> +where
> +
> +* --client means running vdpa app in client mode, in the client mode, QEMU needs
> +  to run as the server mode and take charge of socket file creation.
> +* --iface specifies the path prefix of the UNIX domain socket file, e.g.
> +  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
> +  (n starts from 0).
> +* --interactive means run the vdpa sample in interactive mode, currently 4
> +  internal cmds are supported:
> +
> +  1. help: show help message
> +  2. list: list all available vdpa devices
> +  3. create: create a new vdpa port with socket file and vdpa device address
> +  4. quit: unregister vhost driver and exit the application
> +
> +Take IFCVF driver for example:
> +
> +.. code-block:: console
> +
> +        ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \
> +                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
> +                -- --interactive
> +
> +.. note::
> +    Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices,
> +    and we need to bind vfio-pci to them before running vdpa sample.
> +
> +    * modprobe vfio-pci
> +    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> +        vdpa> list
> +        device id       device address  queue num       supported features
> +        0               0000:06:00.3    1               0x14c238020
> +        1               0000:06:00.4    1               0x14c238020
> +        2               0000:06:00.5    1               0x14c238020
> +
> +        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
> +        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 -cpu host -enable-kvm \
> +       <snip>
> +       -mem-prealloc \
> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> +       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> +    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
> +
> +Live Migration
> +~~~~~~~~~~~~~~
> +vDPA supports cross-backend live migration, user can migrate SW vhost backend
> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
> +the source host with SW vhost VM and B is the destination host with vDPA.
> +
> +1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
> +   in migration-listen mode:
> +
> +.. code-block:: console
> +
> +        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
> +
> +2. Start the migration (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
> +
> +3. Check the status (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) info migrate
> diff --git a/examples/Makefile b/examples/Makefile
> index 481720cb6..356fcb1cd 100644
> --- a/examples/Makefile
> +++ b/examples/Makefile
> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
>   DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
>   endif
>   DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
>   ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
>   DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
>   endif
> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
> new file mode 100644
> index 000000000..42672a2bc
> --- /dev/null
> +++ b/examples/vdpa/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +ifeq ($(RTE_SDK),)
> +$(error "Please define RTE_SDK environment variable")
> +endif
> +
> +# Default target, can be overridden by command line or environment
> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
> +$(info This application can only operate in a linuxapp environment, \
> +please change the definition of the RTE_TARGET environment variable)
> +all:
> +else
> +
> +# binary name
> +APP = vdpa
> +
> +# all source are stored in SRCS-y
> +SRCS-y := main.c
> +
> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -D_GNU_SOURCE
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +
> +include $(RTE_SDK)/mk/rte.extapp.mk
> +
> +endif
> diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
> new file mode 100644
> index 000000000..898286e30
> --- /dev/null
> +++ b/examples/vdpa/main.c
> @@ -0,0 +1,466 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#include <getopt.h>
> +#include <signal.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#include <rte_ethdev.h>
> +#include <rte_malloc.h>
> +#include <rte_vhost.h>
> +#include <rte_vdpa.h>
> +#include <rte_pci.h>
> +
> +#include <cmdline_parse.h>
> +#include <cmdline_socket.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline.h>
> +
> +#define MAX_PATH_LEN 128
> +#define MAX_VDPA_SAMPLE_PORTS 1024
> +#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
> +
> +struct vdpa_port {
> +	char ifname[MAX_PATH_LEN];
> +	int did;
> +	int vid;
> +	uint64_t flags;
> +};
> +
> +struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];

I guess it can be static?

> +
> +static char iface[MAX_PATH_LEN];
> +static int dev_total;
> +static int devcnt;
> +static int interactive;
> +static int client_mode;
> +
> +/* display usage */
> +static void
> +vdpa_usage(const char *prgname)
> +{
> +	printf("Usage: %s [EAL options] -- "
> +				 "	--interactive|-i: run in interactive mode.\n"
> +				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
> +				 "	--client: register a vhost-user socket as client mode.\n",
> +				 prgname);
> +}
> +
> +static int
> +parse_args(int argc, char **argv)
> +{
> +	static const char *short_option = "i";
> +	static struct option long_option[] = {
> +		{"iface", required_argument, NULL, 0},
> +		{"interactive", no_argument, &interactive, 1},
> +		{"client", no_argument, &client_mode, 1},
> +		{NULL, 0, 0, 0},
> +	};
> +	int opt, idx;
> +	char *prgname = argv[0];
> +
> +	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
> +			!= EOF) {
> +		switch (opt) {
> +		case 'i':
> +			printf("Interactive-mode selected\n");
> +			interactive = 1;
> +			break;
> +		/* long options */
> +		case 0:
> +			if (strncmp(long_option[idx].name, "iface",
> +						MAX_PATH_LEN) == 0) {
> +				strncpy(iface, optarg, MAX_PATH_LEN);
> +				printf("iface %s\n", iface);
> +			}
> +			if (!strcmp(long_option[idx].name, "interactive")) {
> +				printf("Interactive-mode selected\n");
> +				interactive = 1;
> +			}
> +			break;
> +
> +		default:
> +			vdpa_usage(prgname);
> +			return -1;
> +		}
> +	}
> +
> +	if (iface[0] == '\0' && interactive == 0) {
> +		vdpa_usage(prgname);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +data_init(void)
> +{
> +	dev_total = rte_vdpa_get_device_num();
> +	if (dev_total <= 0) {
> +		printf("No available vdpa device found\n");
> +		return -1;
> +	}
> +	devcnt = 0;
This isn't needed as it is static.

> +	interactive = 0;
Ditto

> +	memset(iface, 0, sizeof(iface));
Ditto

> +	memset(vports, 0, sizeof(vports));
And this one too when it will be made static.


Appart these small comments, it looks good to me.
When fixed, please apply my:

Reviewed-by: Maxime coquelin <maxime.coquelin@redhat.com>

Thanks,
Maxime

Re: [PATCH v6 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

On 09/27, Maxime Coquelin wrote:
[snip]
>> +	devcnt = 0;
>This isn't needed as it is static.
>
>> +	interactive = 0;
>Ditto
>
>> +	memset(iface, 0, sizeof(iface));
>Ditto
>
>> +	memset(vports, 0, sizeof(vports));
>And this one too when it will be made static.
>
>
>Appart these small comments, it looks good to me.
>When fixed, please apply my:
>
>Reviewed-by: Maxime coquelin <maxime.coquelin@redhat.com>
>

Thanks for your comments, I'll adpot all the static related suggetions.

Thanks,
Xiaolong
>Thanks,
>Maxime

[PATCH v7 0/2] introduce vdpa sample

[Xiaolong Ye]

Hi,

This patchset introduces vdpa sample to demonstrate the vDPA use case.

v7 changes:
* make vports static
* avoid unnecessary static variable initialization

v6 changes:
* improve the document according to Xiao's comments
* fix a typo, PRIu64 -> PRIx64

v5 changes:
* improve print format and correct from "PRIu64" to "PRIx64"
* use "-c 0x2" to better demonstrate app doesn't need to launch dedicated 
  worker threads for vhost enqueue/dequeue operations in vdpa.rst

v4 changes:
* add client mode support
* improve the format to list the vDPA device info and improve the vdpa.rst 
  accordingly
* remove some useless comments
* add introduction in 18.11 release note.


v3 changes:
* list cmd would show queue number and supported features of vdpa devices.
* address Xiao's review comments

v2 changes:

* fix a compilation error reported by Rosen
* improve create cmd in interactive mode and add two new cmds: list, quit
* add application documentation


Xiaolong Ye (2):
  vhost: introduce API to get vDPA device number
  examples/vdpa: introduce a new sample for vDPA

 MAINTAINERS                            |   2 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 115 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 458 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 lib/librte_vhost/rte_vdpa.h            |   3 +
 lib/librte_vhost/rte_vhost_version.map |   1 +
 lib/librte_vhost/vdpa.c                |   6 +
 10 files changed, 635 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

-- 
2.17.1

[PATCH v7 1/2] vhost: introduce API to get vDPA device number

[Xiaolong Ye]

It's used to get number of available registered vDPA devices.

Reviewed-by: Maxime coquelin <maxime.coquelin@redhat.com>
Acked-by: Xiao Wang <xiao.w.wang@intel.com>
Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 lib/librte_vhost/rte_vdpa.h            | 3 +++
 lib/librte_vhost/rte_vhost_version.map | 1 +
 lib/librte_vhost/vdpa.c                | 6 ++++++
 3 files changed, 10 insertions(+)

diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
index 90465ca26..b8223e337 100644
--- a/lib/librte_vhost/rte_vdpa.h
+++ b/lib/librte_vhost/rte_vdpa.h
@@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
 struct rte_vdpa_device * __rte_experimental
 rte_vdpa_get_device(int did);
 
+/* Get current available vdpa device number */
+int __rte_experimental
+rte_vdpa_get_device_num(void);
 #endif /* _RTE_VDPA_H_ */
diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
index da220dd02..ae39b6e21 100644
--- a/lib/librte_vhost/rte_vhost_version.map
+++ b/lib/librte_vhost/rte_vhost_version.map
@@ -67,6 +67,7 @@ EXPERIMENTAL {
 	rte_vdpa_unregister_device;
 	rte_vdpa_find_device_id;
 	rte_vdpa_get_device;
+	rte_vdpa_get_device_num;
 	rte_vhost_driver_attach_vdpa_device;
 	rte_vhost_driver_detach_vdpa_device;
 	rte_vhost_driver_get_vdpa_device_id;
diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
index c82fd4370..c2c5dff1d 100644
--- a/lib/librte_vhost/vdpa.c
+++ b/lib/librte_vhost/vdpa.c
@@ -113,3 +113,9 @@ rte_vdpa_get_device(int did)
 
 	return vdpa_devices[did];
 }
+
+int
+rte_vdpa_get_device_num(void)
+{
+	return vdpa_device_num;
+}
-- 
2.17.1

[PATCH v7 2/2] examples/vdpa: introduce a new sample for vDPA

[Xiaolong Ye]

The vdpa sample application creates vhost-user sockets by using the
vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
virtio ring compatible devices to serve virtio driver directly to enable
datapath acceleration. As vDPA driver can help to set up vhost datapath,
this application doesn't need to launch dedicated worker threads for vhost
enqueue/dequeue operations.

Reviewed-by: Maxime coquelin <maxime.coquelin@redhat.com>
Acked-by: Xiao Wang <xiao.w.wang@intel.com>
Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
---
 MAINTAINERS                            |   2 +
 doc/guides/rel_notes/release_18_11.rst |   8 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 462 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 8 files changed, 642 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

diff --git a/MAINTAINERS b/MAINTAINERS
index 5967c1dd3..5656f18e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
 F: examples/vhost_scsi/
 F: doc/guides/sample_app_ug/vhost_scsi.rst
 F: examples/vhost_crypto/
+F: examples/vdpa/
+F: doc/guides/sample_app_ug/vdpa.rst
 
 Vhost PMD
 M: Maxime Coquelin <maxime.coquelin@redhat.com>
diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index bc9b74ec4..dd53a9ecf 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -67,6 +67,14 @@ New Features
   SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
   vdev_netvsc, tap, and failsafe drivers combination.
 
+* **Add a new sample for vDPA**
+
+  The vdpa sample application creates vhost-user sockets by using the
+  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+  virtio ring compatible devices to serve virtio driver directly to enable
+  datapath acceleration. As vDPA driver can help to set up vhost datapath,
+  this application doesn't need to launch dedicated worker threads for vhost
+  enqueue/dequeue operations.
 
 API Changes
 -----------
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 5bedf4f6f..74b12af85 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -45,6 +45,7 @@ Sample Applications User Guides
     vhost
     vhost_scsi
     vhost_crypto
+    vdpa
     netmap_compatibility
     ip_pipeline
     test_pipeline
diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
new file mode 100644
index 000000000..745f196ca
--- /dev/null
+++ b/doc/guides/sample_app_ug/vdpa.rst
@@ -0,0 +1,120 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+Vdpa Sample Application
+=======================
+
+The vdpa sample application creates vhost-user sockets by using the
+vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+virtio ring compatible devices to serve virtio driver directly to enable
+datapath acceleration. As vDPA driver can help to set up vhost datapath,
+this application doesn't need to launch dedicated worker threads for vhost
+enqueue/dequeue operations.
+
+Testing steps
+-------------
+
+This section shows the steps of how to start VMs with vDPA vhost-user
+backend and verify network connection & live migration.
+
+Build
+~~~~~
+
+To compile the sample application see :doc:`compiling`.
+
+The application is located in the ``vdpa`` sub-directory.
+
+Start the vdpa example
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
+
+where
+
+* --client means running vdpa app in client mode, in the client mode, QEMU needs
+  to run as the server mode and take charge of socket file creation.
+* --iface specifies the path prefix of the UNIX domain socket file, e.g.
+  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
+  (n starts from 0).
+* --interactive means run the vdpa sample in interactive mode, currently 4
+  internal cmds are supported:
+
+  1. help: show help message
+  2. list: list all available vdpa devices
+  3. create: create a new vdpa port with socket file and vdpa device address
+  4. quit: unregister vhost driver and exit the application
+
+Take IFCVF driver for example:
+
+.. code-block:: console
+
+        ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \
+                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
+                -- --interactive
+
+.. note::
+    Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices,
+    and we need to bind vfio-pci to them before running vdpa sample.
+
+    * modprobe vfio-pci
+    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
+
+Then we can create 2 vdpa ports in interactive cmdline.
+
+.. code-block:: console
+
+        vdpa> list
+        device id       device address  queue num       supported features
+        0               0000:06:00.3    1               0x14c238020
+        1               0000:06:00.4    1               0x14c238020
+        2               0000:06:00.5    1               0x14c238020
+
+        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
+        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
+
+.. _vdpa_app_run_vm:
+
+Start the VMs
+~~~~~~~~~~~~~
+
+.. code-block:: console
+
+       qemu-system-x86_64 -cpu host -enable-kvm \
+       <snip>
+       -mem-prealloc \
+       -chardev socket,id=char0,path=<socket_file created in above steps> \
+       -netdev type=vhost-user,id=vdpa,chardev=char0 \
+       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
+
+After the VMs launches, we can login the VMs and configure the ip, verify the
+network connection via ping or netperf.
+
+.. note::
+    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
+
+Live Migration
+~~~~~~~~~~~~~~
+vDPA supports cross-backend live migration, user can migrate SW vhost backend
+VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
+the source host with SW vhost VM and B is the destination host with vDPA.
+
+1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
+   in migration-listen mode:
+
+.. code-block:: console
+
+        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
+
+2. Start the migration (on source host):
+
+.. code-block:: console
+
+        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
+
+3. Check the status (on source host):
+
+.. code-block:: console
+
+        A: (qemu) info migrate
diff --git a/examples/Makefile b/examples/Makefile
index 481720cb6..356fcb1cd 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
-DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
+DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
 ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
 endif
diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
new file mode 100644
index 000000000..42672a2bc
--- /dev/null
+++ b/examples/vdpa/Makefile
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
+$(info This application can only operate in a linuxapp environment, \
+please change the definition of the RTE_TARGET environment variable)
+all:
+else
+
+# binary name
+APP = vdpa
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -D_GNU_SOURCE
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
new file mode 100644
index 000000000..b8055287f
--- /dev/null
+++ b/examples/vdpa/main.c
@@ -0,0 +1,462 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <getopt.h>
+#include <signal.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_ethdev.h>
+#include <rte_malloc.h>
+#include <rte_vhost.h>
+#include <rte_vdpa.h>
+#include <rte_pci.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#define MAX_PATH_LEN 128
+#define MAX_VDPA_SAMPLE_PORTS 1024
+#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
+
+struct vdpa_port {
+	char ifname[MAX_PATH_LEN];
+	int did;
+	int vid;
+	uint64_t flags;
+};
+
+static struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
+
+static char iface[MAX_PATH_LEN];
+static int dev_total;
+static int devcnt;
+static int interactive;
+static int client_mode;
+
+/* display usage */
+static void
+vdpa_usage(const char *prgname)
+{
+	printf("Usage: %s [EAL options] -- "
+				 "	--interactive|-i: run in interactive mode.\n"
+				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				 "	--client: register a vhost-user socket as client mode.\n",
+				 prgname);
+}
+
+static int
+parse_args(int argc, char **argv)
+{
+	static const char *short_option = "i";
+	static struct option long_option[] = {
+		{"iface", required_argument, NULL, 0},
+		{"interactive", no_argument, &interactive, 1},
+		{"client", no_argument, &client_mode, 1},
+		{NULL, 0, 0, 0},
+	};
+	int opt, idx;
+	char *prgname = argv[0];
+
+	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
+			!= EOF) {
+		switch (opt) {
+		case 'i':
+			printf("Interactive-mode selected\n");
+			interactive = 1;
+			break;
+		/* long options */
+		case 0:
+			if (strncmp(long_option[idx].name, "iface",
+						MAX_PATH_LEN) == 0) {
+				strncpy(iface, optarg, MAX_PATH_LEN);
+				printf("iface %s\n", iface);
+			}
+			if (!strcmp(long_option[idx].name, "interactive")) {
+				printf("Interactive-mode selected\n");
+				interactive = 1;
+			}
+			break;
+
+		default:
+			vdpa_usage(prgname);
+			return -1;
+		}
+	}
+
+	if (iface[0] == '\0' && interactive == 0) {
+		vdpa_usage(prgname);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+data_init(void)
+{
+	dev_total = rte_vdpa_get_device_num();
+	if (dev_total <= 0) {
+		printf("No available vdpa device found\n");
+		return -1;
+	}
+	return 0;
+}
+
+static int
+new_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
+			printf("\nnew port %s, did: %d\n",
+					ifname, vports[i].did);
+			vports[i].vid = vid;
+			break;
+		}
+	}
+
+	if (i >= MAX_VDPA_SAMPLE_PORTS)
+		return -1;
+
+	return 0;
+}
+
+static void
+destroy_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strcmp(ifname, vports[i].ifname) == 0) {
+			printf("\ndestroy port %s, did: %d\n",
+					ifname, vports[i].did);
+			break;
+		}
+	}
+}
+
+static const struct vhost_device_ops vdpa_sample_devops = {
+	.new_device = new_device,
+	.destroy_device = destroy_device,
+};
+
+static int
+start_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+	int did = vport->did;
+
+	if (client_mode)
+		vport->flags |= RTE_VHOST_USER_CLIENT;
+
+	if (access(socket_path, F_OK) != -1 && !client_mode) {
+		RTE_LOG(ERR, VDPA,
+			"%s exists, please remove it or specify another file and try again.\n",
+			socket_path);
+		return -1;
+	}
+	ret = rte_vhost_driver_register(socket_path, vport->flags);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_callback_register(socket_path,
+			&vdpa_sample_devops);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver ops failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"attach vdpa device failed: %s\n",
+			socket_path);
+
+	if (rte_vhost_driver_start(socket_path) < 0)
+		rte_exit(EXIT_FAILURE,
+			"start vhost driver failed: %s\n",
+			socket_path);
+	return 0;
+}
+
+static void
+close_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+
+	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"detach vdpa device failed: %s\n",
+				socket_path);
+
+	ret = rte_vhost_driver_unregister(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"Fail to unregister vhost driver for %s.\n",
+				socket_path);
+}
+
+static void
+vdpa_sample_quit(void)
+{
+	int i;
+	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
+		if (vports[i].ifname[0] != '\0')
+			close_vdpa(&vports[i]);
+	}
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\nSignal %d received, preparing to exit...\n", signum);
+		vdpa_sample_quit();
+		exit(0);
+	}
+}
+
+/* interactive cmds */
+
+/* *** Help command with introduction. *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	cmdline_printf(
+		cl,
+		"\n"
+		"The following commands are currently available:\n\n"
+		"Control:\n"
+		"    help                                      : Show interactive instructions.\n"
+		"    list                                      : list all available vdpa devices.\n"
+		"    create <socket file> <vdev addr>          : create a new vdpa port.\n"
+		"    quit                                      : exit vdpa sample app.\n"
+	);
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show help",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/* *** List all available vdpa devices *** */
+struct cmd_list_result {
+	cmdline_fixed_string_t action;
+};
+
+static void cmd_list_vdpa_devices_parsed(
+		__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	uint32_t queue_num;
+	uint64_t features;
+	struct rte_vdpa_device *vdev;
+	struct rte_pci_addr addr;
+
+	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
+	for (did = 0; did < dev_total; did++) {
+		vdev = rte_vdpa_get_device(did);
+		if (!vdev)
+			continue;
+		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa queue number "
+				"for device id %d.\n", did);
+			continue;
+		}
+		if (vdev->ops->get_features(did, &features) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa features "
+				"for device id %d.\n", did);
+			continue;
+		}
+		addr = vdev->addr.pci_addr;
+		cmdline_printf(cl,
+			"%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIx64 "\n",
+			did, addr.domain, addr.bus, addr.devid,
+			addr.function, queue_num, features);
+	}
+}
+
+cmdline_parse_token_string_t cmd_action_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
+
+cmdline_parse_inst_t cmd_list_vdpa_devices = {
+	.f = cmd_list_vdpa_devices_parsed,
+	.data = NULL,
+	.help_str = "list all available vdpa devices",
+	.tokens = {
+		(void *)&cmd_action_list,
+		NULL,
+	},
+};
+
+/* *** Create new vdpa port *** */
+struct cmd_create_result {
+	cmdline_fixed_string_t action;
+	cmdline_fixed_string_t socket_path;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_create_vdpa_port_parsed(void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	struct cmd_create_result *res = parsed_result;
+	struct rte_vdpa_dev_addr addr;
+
+	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
+	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
+		cmdline_printf(cl, "Unable to parse the given bdf.\n");
+		return;
+	}
+	addr.type = PCI_ADDR;
+	did = rte_vdpa_find_device_id(&addr);
+	if (did < 0) {
+		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
+		return;
+	}
+
+	vports[devcnt].did = did;
+
+	if (start_vdpa(&vports[devcnt]) == 0)
+		devcnt++;
+}
+
+cmdline_parse_token_string_t cmd_action_create =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
+cmdline_parse_token_string_t cmd_socket_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
+cmdline_parse_token_string_t cmd_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_create_vdpa_port = {
+	.f = cmd_create_vdpa_port_parsed,
+	.data = NULL,
+	.help_str = "create a new vdpa port",
+	.tokens = {
+		(void *)&cmd_action_create,
+		(void *)&cmd_socket_path,
+		(void *)&cmd_bdf,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	vdpa_sample_quit();
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "quit: exit application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
+	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	NULL,
+};
+
+int
+main(int argc, char *argv[])
+{
+	char ch;
+	int i;
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "eal init failed\n");
+	argc -= ret;
+	argv += ret;
+
+	data_init();
+
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	ret = parse_args(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "invalid argument\n");
+
+	if (interactive == 1) {
+		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
+		if (cl == NULL)
+			rte_panic("Cannot create cmdline instance\n");
+		cmdline_interact(cl);
+		cmdline_stdin_exit(cl);
+	} else {
+		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
+				i++) {
+			vports[i].did = i;
+			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
+					iface, i);
+
+			start_vdpa(&vports[i]);
+		}
+
+		printf("enter \'q\' to quit\n");
+		while (scanf("%c", &ch)) {
+			if (ch == 'q')
+				break;
+			while (ch != '\n') {
+				if (scanf("%c", &ch))
+					printf("%c", ch);
+			}
+			printf("enter \'q\' to quit\n");
+		}
+		vdpa_sample_quit();
+	}
+
+	return 0;
+}
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
new file mode 100644
index 000000000..2e38a069f
--- /dev/null
+++ b/examples/vdpa/meson.build
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'vhost'
+allow_experimental_apis = true
+sources = files(
+	'main.c'
+)
\ No newline at end of file
-- 
2.17.1

Re: [PATCH v7 2/2] examples/vdpa: introduce a new sample for vDPA

[Maxime Coquelin]

On 09/28/2018 01:23 PM, Xiaolong Ye wrote:
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Reviewed-by: Maxime coquelin <maxime.coquelin@redhat.com>
> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> ---
>   MAINTAINERS                            |   2 +
>   doc/guides/rel_notes/release_18_11.rst |   8 +
>   doc/guides/sample_app_ug/index.rst     |   1 +
>   doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
>   examples/Makefile                      |   2 +-
>   examples/vdpa/Makefile                 |  32 ++
>   examples/vdpa/main.c                   | 462 +++++++++++++++++++++++++
>   examples/vdpa/meson.build              |  16 +
>   8 files changed, 642 insertions(+), 1 deletion(-)
>   create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>   create mode 100644 examples/vdpa/Makefile
>   create mode 100644 examples/vdpa/main.c
>   create mode 100644 examples/vdpa/meson.build
> 
> diff --git a/MAINTAINERS b/MAINTAINERS
> index 5967c1dd3..5656f18e8 100644
> --- a/MAINTAINERS
> +++ b/MAINTAINERS
> @@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
>   F: examples/vhost_scsi/
>   F: doc/guides/sample_app_ug/vhost_scsi.rst
>   F: examples/vhost_crypto/
> +F: examples/vdpa/
> +F: doc/guides/sample_app_ug/vdpa.rst
>   
>   Vhost PMD
>   M: Maxime Coquelin <maxime.coquelin@redhat.com>
> diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
> index bc9b74ec4..dd53a9ecf 100644
> --- a/doc/guides/rel_notes/release_18_11.rst
> +++ b/doc/guides/rel_notes/release_18_11.rst
> @@ -67,6 +67,14 @@ New Features
>     SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
>     vdev_netvsc, tap, and failsafe drivers combination.
>   
> +* **Add a new sample for vDPA**
> +
> +  The vdpa sample application creates vhost-user sockets by using the
> +  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +  virtio ring compatible devices to serve virtio driver directly to enable
> +  datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +  this application doesn't need to launch dedicated worker threads for vhost
> +  enqueue/dequeue operations.
>   
>   API Changes
>   -----------
> diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
> index 5bedf4f6f..74b12af85 100644
> --- a/doc/guides/sample_app_ug/index.rst
> +++ b/doc/guides/sample_app_ug/index.rst
> @@ -45,6 +45,7 @@ Sample Applications User Guides
>       vhost
>       vhost_scsi
>       vhost_crypto
> +    vdpa
>       netmap_compatibility
>       ip_pipeline
>       test_pipeline
> diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
> new file mode 100644
> index 000000000..745f196ca
> --- /dev/null
> +++ b/doc/guides/sample_app_ug/vdpa.rst
> @@ -0,0 +1,120 @@
> +..  SPDX-License-Identifier: BSD-3-Clause
> +    Copyright(c) 2018 Intel Corporation.
> +
> +Vdpa Sample Application
> +=======================
> +
> +The vdpa sample application creates vhost-user sockets by using the
> +vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> +virtio ring compatible devices to serve virtio driver directly to enable
> +datapath acceleration. As vDPA driver can help to set up vhost datapath,
> +this application doesn't need to launch dedicated worker threads for vhost
> +enqueue/dequeue operations.
> +
> +Testing steps
> +-------------
> +
> +This section shows the steps of how to start VMs with vDPA vhost-user
> +backend and verify network connection & live migration.
> +
> +Build
> +~~~~~
> +
> +To compile the sample application see :doc:`compiling`.
> +
> +The application is located in the ``vdpa`` sub-directory.
> +
> +Start the vdpa example
> +~~~~~~~~~~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
> +
> +where
> +
> +* --client means running vdpa app in client mode, in the client mode, QEMU needs
> +  to run as the server mode and take charge of socket file creation.
> +* --iface specifies the path prefix of the UNIX domain socket file, e.g.
> +  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
> +  (n starts from 0).
> +* --interactive means run the vdpa sample in interactive mode, currently 4
> +  internal cmds are supported:
> +
> +  1. help: show help message
> +  2. list: list all available vdpa devices
> +  3. create: create a new vdpa port with socket file and vdpa device address
> +  4. quit: unregister vhost driver and exit the application
> +
> +Take IFCVF driver for example:
> +
> +.. code-block:: console
> +
> +        ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \
> +                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
> +                -- --interactive
> +
> +.. note::
> +    Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices,
> +    and we need to bind vfio-pci to them before running vdpa sample.
> +
> +    * modprobe vfio-pci
> +    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
> +
> +Then we can create 2 vdpa ports in interactive cmdline.
> +
> +.. code-block:: console
> +
> +        vdpa> list
> +        device id       device address  queue num       supported features
> +        0               0000:06:00.3    1               0x14c238020
> +        1               0000:06:00.4    1               0x14c238020
> +        2               0000:06:00.5    1               0x14c238020
> +
> +        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
> +        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
> +
> +.. _vdpa_app_run_vm:
> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 -cpu host -enable-kvm \
> +       <snip>
> +       -mem-prealloc \
> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> +       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
> +
> +After the VMs launches, we can login the VMs and configure the ip, verify the
> +network connection via ping or netperf.
> +
> +.. note::
> +    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
> +
> +Live Migration
> +~~~~~~~~~~~~~~
> +vDPA supports cross-backend live migration, user can migrate SW vhost backend
> +VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
> +the source host with SW vhost VM and B is the destination host with vDPA.
> +
> +1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
> +   in migration-listen mode:
> +
> +.. code-block:: console
> +
> +        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
> +
> +2. Start the migration (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
> +
> +3. Check the status (on source host):
> +
> +.. code-block:: console
> +
> +        A: (qemu) info migrate
> diff --git a/examples/Makefile b/examples/Makefile
> index 481720cb6..356fcb1cd 100644
> --- a/examples/Makefile
> +++ b/examples/Makefile
> @@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
>   DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
>   endif
>   DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
> -DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
> +DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
>   ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
>   DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
>   endif
> diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
> new file mode 100644
> index 000000000..42672a2bc
> --- /dev/null
> +++ b/examples/vdpa/Makefile
> @@ -0,0 +1,32 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +ifeq ($(RTE_SDK),)
> +$(error "Please define RTE_SDK environment variable")
> +endif
> +
> +# Default target, can be overridden by command line or environment
> +RTE_TARGET ?= x86_64-native-linuxapp-gcc
> +
> +include $(RTE_SDK)/mk/rte.vars.mk
> +
> +ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
> +$(info This application can only operate in a linuxapp environment, \
> +please change the definition of the RTE_TARGET environment variable)
> +all:
> +else
> +
> +# binary name
> +APP = vdpa
> +
> +# all source are stored in SRCS-y
> +SRCS-y := main.c
> +
> +CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
> +CFLAGS += $(WERROR_FLAGS)
> +CFLAGS += -D_GNU_SOURCE
> +CFLAGS += -DALLOW_EXPERIMENTAL_API
> +
> +include $(RTE_SDK)/mk/rte.extapp.mk
> +
> +endif
> diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
> new file mode 100644
> index 000000000..b8055287f
> --- /dev/null
> +++ b/examples/vdpa/main.c
> @@ -0,0 +1,462 @@
> +/* SPDX-License-Identifier: BSD-3-Clause
> + * Copyright(c) 2018 Intel Corporation
> + */
> +
> +#include <getopt.h>
> +#include <signal.h>
> +#include <stdint.h>
> +#include <string.h>
> +#include <unistd.h>
> +
> +#include <rte_ethdev.h>
> +#include <rte_malloc.h>
> +#include <rte_vhost.h>
> +#include <rte_vdpa.h>
> +#include <rte_pci.h>
> +
> +#include <cmdline_parse.h>
> +#include <cmdline_socket.h>
> +#include <cmdline_parse_string.h>
> +#include <cmdline.h>
> +
> +#define MAX_PATH_LEN 128
> +#define MAX_VDPA_SAMPLE_PORTS 1024
> +#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
> +
> +struct vdpa_port {
> +	char ifname[MAX_PATH_LEN];
> +	int did;
> +	int vid;
> +	uint64_t flags;
> +};
> +
> +static struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
> +
> +static char iface[MAX_PATH_LEN];
> +static int dev_total;
> +static int devcnt;
> +static int interactive;
> +static int client_mode;
> +
> +/* display usage */
> +static void
> +vdpa_usage(const char *prgname)
> +{
> +	printf("Usage: %s [EAL options] -- "
> +				 "	--interactive|-i: run in interactive mode.\n"
> +				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
> +				 "	--client: register a vhost-user socket as client mode.\n",
> +				 prgname);
> +}
> +
> +static int
> +parse_args(int argc, char **argv)
> +{
> +	static const char *short_option = "i";
> +	static struct option long_option[] = {
> +		{"iface", required_argument, NULL, 0},
> +		{"interactive", no_argument, &interactive, 1},
> +		{"client", no_argument, &client_mode, 1},
> +		{NULL, 0, 0, 0},
> +	};
> +	int opt, idx;
> +	char *prgname = argv[0];
> +
> +	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
> +			!= EOF) {
> +		switch (opt) {
> +		case 'i':
> +			printf("Interactive-mode selected\n");
> +			interactive = 1;
> +			break;
> +		/* long options */
> +		case 0:
> +			if (strncmp(long_option[idx].name, "iface",
> +						MAX_PATH_LEN) == 0) {
> +				strncpy(iface, optarg, MAX_PATH_LEN);
> +				printf("iface %s\n", iface);
> +			}
> +			if (!strcmp(long_option[idx].name, "interactive")) {
> +				printf("Interactive-mode selected\n");
> +				interactive = 1;
> +			}
> +			break;
> +
> +		default:
> +			vdpa_usage(prgname);
> +			return -1;
> +		}
> +	}
> +
> +	if (iface[0] == '\0' && interactive == 0) {
> +		vdpa_usage(prgname);
> +		return -1;
> +	}
> +
> +	return 0;
> +}
> +
> +static int
> +data_init(void)
> +{
> +	dev_total = rte_vdpa_get_device_num();
> +	if (dev_total <= 0) {
> +		printf("No available vdpa device found\n");
> +		return -1;
> +	}
> +	return 0;
> +}
> +
> +static int
> +new_device(int vid)
> +{
> +	char ifname[MAX_PATH_LEN];
> +	int i;
> +
> +	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
> +	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
> +		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
> +			printf("\nnew port %s, did: %d\n",
> +					ifname, vports[i].did);
> +			vports[i].vid = vid;
> +			break;
> +		}
> +	}
> +
> +	if (i >= MAX_VDPA_SAMPLE_PORTS)
> +		return -1;
> +
> +	return 0;
> +}
> +
> +static void
> +destroy_device(int vid)
> +{
> +	char ifname[MAX_PATH_LEN];
> +	int i;
> +
> +	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
> +	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
> +		if (strcmp(ifname, vports[i].ifname) == 0) {
> +			printf("\ndestroy port %s, did: %d\n",
> +					ifname, vports[i].did);
> +			break;
> +		}
> +	}
> +}
> +
> +static const struct vhost_device_ops vdpa_sample_devops = {
> +	.new_device = new_device,
> +	.destroy_device = destroy_device,
> +};
> +
> +static int
> +start_vdpa(struct vdpa_port *vport)
> +{
> +	int ret;
> +	char *socket_path = vport->ifname;
> +	int did = vport->did;
> +
> +	if (client_mode)
> +		vport->flags |= RTE_VHOST_USER_CLIENT;
> +
> +	if (access(socket_path, F_OK) != -1 && !client_mode) {
> +		RTE_LOG(ERR, VDPA,
> +			"%s exists, please remove it or specify another file and try again.\n",
> +			socket_path);
> +		return -1;
> +	}
> +	ret = rte_vhost_driver_register(socket_path, vport->flags);
> +	if (ret != 0)
> +		rte_exit(EXIT_FAILURE,
> +			"register driver failed: %s\n",
> +			socket_path);
> +
> +	ret = rte_vhost_driver_callback_register(socket_path,
> +			&vdpa_sample_devops);
> +	if (ret != 0)
> +		rte_exit(EXIT_FAILURE,
> +			"register driver ops failed: %s\n",
> +			socket_path);
> +
> +	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
> +	if (ret != 0)
> +		rte_exit(EXIT_FAILURE,
> +			"attach vdpa device failed: %s\n",
> +			socket_path);
> +
> +	if (rte_vhost_driver_start(socket_path) < 0)
> +		rte_exit(EXIT_FAILURE,
> +			"start vhost driver failed: %s\n",
> +			socket_path);
> +	return 0;
> +}
> +
> +static void
> +close_vdpa(struct vdpa_port *vport)
> +{
> +	int ret;
> +	char *socket_path = vport->ifname;
> +
> +	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
> +	if (ret != 0)
> +		RTE_LOG(ERR, VDPA,
> +				"detach vdpa device failed: %s\n",
> +				socket_path);
> +
> +	ret = rte_vhost_driver_unregister(socket_path);
> +	if (ret != 0)
> +		RTE_LOG(ERR, VDPA,
> +				"Fail to unregister vhost driver for %s.\n",
> +				socket_path);
> +}
> +
> +static void
> +vdpa_sample_quit(void)
> +{
> +	int i;
> +	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
> +		if (vports[i].ifname[0] != '\0')
> +			close_vdpa(&vports[i]);
> +	}
> +}
> +
> +static void
> +signal_handler(int signum)
> +{
> +	if (signum == SIGINT || signum == SIGTERM) {
> +		printf("\nSignal %d received, preparing to exit...\n", signum);
> +		vdpa_sample_quit();
> +		exit(0);
> +	}
> +}
> +
> +/* interactive cmds */
> +
> +/* *** Help command with introduction. *** */
> +struct cmd_help_result {
> +	cmdline_fixed_string_t help;
> +};
> +
> +static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	cmdline_printf(
> +		cl,
> +		"\n"
> +		"The following commands are currently available:\n\n"
> +		"Control:\n"
> +		"    help                                      : Show interactive instructions.\n"
> +		"    list                                      : list all available vdpa devices.\n"
> +		"    create <socket file> <vdev addr>          : create a new vdpa port.\n"
> +		"    quit                                      : exit vdpa sample app.\n"
> +	);
> +}
> +
> +cmdline_parse_token_string_t cmd_help_help =
> +	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
> +
> +cmdline_parse_inst_t cmd_help = {
> +	.f = cmd_help_parsed,
> +	.data = NULL,
> +	.help_str = "show help",
> +	.tokens = {
> +		(void *)&cmd_help_help,
> +		NULL,
> +	},
> +};
> +
> +/* *** List all available vdpa devices *** */
> +struct cmd_list_result {
> +	cmdline_fixed_string_t action;
> +};
> +
> +static void cmd_list_vdpa_devices_parsed(
> +		__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	int did;
> +	uint32_t queue_num;
> +	uint64_t features;
> +	struct rte_vdpa_device *vdev;
> +	struct rte_pci_addr addr;
> +
> +	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
> +	for (did = 0; did < dev_total; did++) {
> +		vdev = rte_vdpa_get_device(did);
> +		if (!vdev)
> +			continue;
> +		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa queue number "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		if (vdev->ops->get_features(did, &features) < 0) {
> +			RTE_LOG(ERR, VDPA,
> +				"failed to get vdpa features "
> +				"for device id %d.\n", did);
> +			continue;
> +		}
> +		addr = vdev->addr.pci_addr;
> +		cmdline_printf(cl,
> +			"%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIx64 "\n",
> +			did, addr.domain, addr.bus, addr.devid,
> +			addr.function, queue_num, features);
> +	}
> +}
> +
> +cmdline_parse_token_string_t cmd_action_list =
> +	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
> +
> +cmdline_parse_inst_t cmd_list_vdpa_devices = {
> +	.f = cmd_list_vdpa_devices_parsed,
> +	.data = NULL,
> +	.help_str = "list all available vdpa devices",
> +	.tokens = {
> +		(void *)&cmd_action_list,
> +		NULL,
> +	},
> +};
> +
> +/* *** Create new vdpa port *** */
> +struct cmd_create_result {
> +	cmdline_fixed_string_t action;
> +	cmdline_fixed_string_t socket_path;
> +	cmdline_fixed_string_t bdf;
> +};
> +
> +static void cmd_create_vdpa_port_parsed(void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	int did;
> +	struct cmd_create_result *res = parsed_result;
> +	struct rte_vdpa_dev_addr addr;
> +
> +	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
> +	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
> +		cmdline_printf(cl, "Unable to parse the given bdf.\n");
> +		return;
> +	}
> +	addr.type = PCI_ADDR;
> +	did = rte_vdpa_find_device_id(&addr);
> +	if (did < 0) {
> +		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
> +		return;
> +	}
> +
> +	vports[devcnt].did = did;
> +
> +	if (start_vdpa(&vports[devcnt]) == 0)
> +		devcnt++;
> +}
> +
> +cmdline_parse_token_string_t cmd_action_create =
> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
> +cmdline_parse_token_string_t cmd_socket_path =
> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
> +cmdline_parse_token_string_t cmd_bdf =
> +	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
> +
> +cmdline_parse_inst_t cmd_create_vdpa_port = {
> +	.f = cmd_create_vdpa_port_parsed,
> +	.data = NULL,
> +	.help_str = "create a new vdpa port",
> +	.tokens = {
> +		(void *)&cmd_action_create,
> +		(void *)&cmd_socket_path,
> +		(void *)&cmd_bdf,
> +		NULL,
> +	},
> +};
> +
> +/* *** QUIT *** */
> +struct cmd_quit_result {
> +	cmdline_fixed_string_t quit;
> +};
> +
> +static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
> +		struct cmdline *cl,
> +		__attribute__((unused)) void *data)
> +{
> +	vdpa_sample_quit();
> +	cmdline_quit(cl);
> +}
> +
> +cmdline_parse_token_string_t cmd_quit_quit =
> +	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
> +
> +cmdline_parse_inst_t cmd_quit = {
> +	.f = cmd_quit_parsed,
> +	.data = NULL,
> +	.help_str = "quit: exit application",
> +	.tokens = {
> +		(void *)&cmd_quit_quit,
> +		NULL,
> +	},
> +};
> +cmdline_parse_ctx_t main_ctx[] = {
> +	(cmdline_parse_inst_t *)&cmd_help,
> +	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
> +	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
> +	(cmdline_parse_inst_t *)&cmd_quit,
> +	NULL,
> +};
> +
> +int
> +main(int argc, char *argv[])
> +{
> +	char ch;
> +	int i;
> +	int ret;
> +	struct cmdline *cl;
> +
> +	ret = rte_eal_init(argc, argv);
> +	if (ret < 0)
> +		rte_exit(EXIT_FAILURE, "eal init failed\n");
> +	argc -= ret;
> +	argv += ret;
> +
> +	data_init();

You need to check return from data_init().
If this is fine for you, I can change to below code when applying:

ret = data_init();
if (ret < 0)
     rte_exit(EXIT_FAILURE, "data init failed\n");

> +
> +	signal(SIGINT, signal_handler);
> +	signal(SIGTERM, signal_handler);
> +
> +	ret = parse_args(argc, argv);
> +	if (ret < 0)
> +		rte_exit(EXIT_FAILURE, "invalid argument\n");
> +
> +	if (interactive == 1) {
> +		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
> +		if (cl == NULL)
> +			rte_panic("Cannot create cmdline instance\n");
> +		cmdline_interact(cl);
> +		cmdline_stdin_exit(cl);
> +	} else {
> +		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
> +				i++) {
> +			vports[i].did = i;
> +			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
> +					iface, i);
> +
> +			start_vdpa(&vports[i]);
> +		}
> +
> +		printf("enter \'q\' to quit\n");
> +		while (scanf("%c", &ch)) {
> +			if (ch == 'q')
> +				break;
> +			while (ch != '\n') {
> +				if (scanf("%c", &ch))
> +					printf("%c", ch);
> +			}
> +			printf("enter \'q\' to quit\n");
> +		}
> +		vdpa_sample_quit();
> +	}
> +
> +	return 0;
> +}
> diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
> new file mode 100644
> index 000000000..2e38a069f
> --- /dev/null
> +++ b/examples/vdpa/meson.build
> @@ -0,0 +1,16 @@
> +# SPDX-License-Identifier: BSD-3-Clause
> +# Copyright(c) 2018 Intel Corporation
> +
> +# meson file, for building this example as part of a main DPDK build.
> +#
> +# To build this example as a standalone application with an already-installed
> +# DPDK instance, use 'make'
> +
> +if host_machine.system() != 'linux'
> +	build = false
> +endif
> +deps += 'vhost'
> +allow_experimental_apis = true
> +sources = files(
> +	'main.c'
> +)
> \ No newline at end of file
> 

Re: [PATCH v7 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

On 09/28, Maxime Coquelin wrote:
>> +	ret = rte_eal_init(argc, argv);
>> +	if (ret < 0)
>> +		rte_exit(EXIT_FAILURE, "eal init failed\n");
>> +	argc -= ret;
>> +	argv += ret;
>> +
>> +	data_init();
>
>You need to check return from data_init().
>If this is fine for you, I can change to below code when applying:
>
>ret = data_init();
>if (ret < 0)
>    rte_exit(EXIT_FAILURE, "data init failed\n");
>
> 

I'm fine with this change.

Thanks,
Xiaolong

Re: [PATCH v7 2/2] examples/vdpa: introduce a new sample for vDPA

[Maxime Coquelin]

On 09/28/2018 04:33 PM, Ye Xiaolong wrote:
> On 09/28, Maxime Coquelin wrote:
>>> +	ret = rte_eal_init(argc, argv);
>>> +	if (ret < 0)
>>> +		rte_exit(EXIT_FAILURE, "eal init failed\n");
>>> +	argc -= ret;
>>> +	argv += ret;
>>> +
>>> +	data_init();
>>
>> You need to check return from data_init().
>> If this is fine for you, I can change to below code when applying:
>>
>> ret = data_init();
>> if (ret < 0)
>>     rte_exit(EXIT_FAILURE, "data init failed\n");
>>
>>
> 
> I'm fine with this change.

Or even better, get rid of data_init and
call rte_vdpa_get_device_num(); in main directly.

> Thanks,
> Xiaolong
> 

Re: [PATCH v7 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

On 09/28, Maxime Coquelin wrote:
>
>
>On 09/28/2018 04:33 PM, Ye Xiaolong wrote:
>> On 09/28, Maxime Coquelin wrote:
>> > > +	ret = rte_eal_init(argc, argv);
>> > > +	if (ret < 0)
>> > > +		rte_exit(EXIT_FAILURE, "eal init failed\n");
>> > > +	argc -= ret;
>> > > +	argv += ret;
>> > > +
>> > > +	data_init();
>> > 
>> > You need to check return from data_init().
>> > If this is fine for you, I can change to below code when applying:
>> > 
>> > ret = data_init();
>> > if (ret < 0)
>> >     rte_exit(EXIT_FAILURE, "data init failed\n");
>> > 
>> > 
>> 
>> I'm fine with this change.
>
>Or even better, get rid of data_init and
>call rte_vdpa_get_device_num(); in main directly.

Sounds better, as we've removed unnecessary static variable initialization in  
data_init, it does nothing but call rte_vdpa_get_device_num, we can remove this
wrap. Would you do this change when applying or you perfer I send a new 
revision?

Thanks,
Xiaolong


>
>> Thanks,
>> Xiaolong
>> 

Re: [PATCH v7 2/2] examples/vdpa: introduce a new sample for vDPA

[Maxime Coquelin]

On 09/28/2018 05:17 PM, Ye Xiaolong wrote:
> On 09/28, Maxime Coquelin wrote:
>>
>>
>> On 09/28/2018 04:33 PM, Ye Xiaolong wrote:
>>> On 09/28, Maxime Coquelin wrote:
>>>>> +	ret = rte_eal_init(argc, argv);
>>>>> +	if (ret < 0)
>>>>> +		rte_exit(EXIT_FAILURE, "eal init failed\n");
>>>>> +	argc -= ret;
>>>>> +	argv += ret;
>>>>> +
>>>>> +	data_init();
>>>>
>>>> You need to check return from data_init().
>>>> If this is fine for you, I can change to below code when applying:
>>>>
>>>> ret = data_init();
>>>> if (ret < 0)
>>>>      rte_exit(EXIT_FAILURE, "data init failed\n");
>>>>
>>>>
>>>
>>> I'm fine with this change.
>>
>> Or even better, get rid of data_init and
>> call rte_vdpa_get_device_num(); in main directly.
> 
> Sounds better, as we've removed unnecessary static variable initialization in
> data_init, it does nothing but call rte_vdpa_get_device_num, we can remove this
> wrap. Would you do this change when applying or you perfer I send a new
> revision?

Please send a new revision, and in the commit message,
fix first letter of my family name with an upper case,
and move my R-b below your sign-off.

Thanks!
Maxime

> Thanks,
> Xiaolong
> 
> 
>>
>>> Thanks,
>>> Xiaolong
>>>

Re: [PATCH v7 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

On 09/28, Maxime Coquelin wrote:
>
>
>On 09/28/2018 05:17 PM, Ye Xiaolong wrote:
>> On 09/28, Maxime Coquelin wrote:
>> > 
>> > 
>> > On 09/28/2018 04:33 PM, Ye Xiaolong wrote:
>> > > On 09/28, Maxime Coquelin wrote:
>> > > > > +	ret = rte_eal_init(argc, argv);
>> > > > > +	if (ret < 0)
>> > > > > +		rte_exit(EXIT_FAILURE, "eal init failed\n");
>> > > > > +	argc -= ret;
>> > > > > +	argv += ret;
>> > > > > +
>> > > > > +	data_init();
>> > > > 
>> > > > You need to check return from data_init().
>> > > > If this is fine for you, I can change to below code when applying:
>> > > > 
>> > > > ret = data_init();
>> > > > if (ret < 0)
>> > > >      rte_exit(EXIT_FAILURE, "data init failed\n");
>> > > > 
>> > > > 
>> > > 
>> > > I'm fine with this change.
>> > 
>> > Or even better, get rid of data_init and
>> > call rte_vdpa_get_device_num(); in main directly.
>> 
>> Sounds better, as we've removed unnecessary static variable initialization in
>> data_init, it does nothing but call rte_vdpa_get_device_num, we can remove this
>> wrap. Would you do this change when applying or you perfer I send a new
>> revision?
>
>Please send a new revision, and in the commit message,
>fix first letter of my family name with an upper case,
>and move my R-b below your sign-off.

Got it.

Thanks,
Xiaolong
>
>Thanks!
>Maxime
>
>> Thanks,
>> Xiaolong
>> 
>> 
>> > 
>> > > Thanks,
>> > > Xiaolong
>> > > 

[PATCH v8 0/2] introduce vdpa sample

[Xiaolong Ye]

Hi,

This patchset introduces vdpa sample to demonstrate the vDPA use case.

v8 changes:
* move body of data_init to the main function
* adjust Reviewed-by/Acked-by tag postion

v7 changes:
* make vports static
* avoid unnecessary static variable initialization

v6 changes:
* improve the document according to Xiao's comments
* fix a typo, PRIu64 -> PRIx64

v5 changes:
* improve print format and correct from "PRIu64" to "PRIx64"
* use "-c 0x2" to better demonstrate app doesn't need to launch dedicated 
  worker threads for vhost enqueue/dequeue operations in vdpa.rst

v4 changes:
* add client mode support
* improve the format to list the vDPA device info and improve the vdpa.rst 
  accordingly
* remove some useless comments
* add introduction in 18.11 release note.


v3 changes:
* list cmd would show queue number and supported features of vdpa devices.
* address Xiao's review comments

v2 changes:

* fix a compilation error reported by Rosen
* improve create cmd in interactive mode and add two new cmds: list, quit
* add application documentation


Xiaolong Ye (2):
  vhost: introduce API to get vDPA device number
  examples/vdpa: introduce a new sample for vDPA

 MAINTAINERS                            |   2 +
 doc/guides/rel_notes/release_18_11.rst |   8 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 453 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 lib/librte_vhost/rte_vdpa.h            |   3 +
 lib/librte_vhost/rte_vhost_version.map |   1 +
 lib/librte_vhost/vdpa.c                |   6 +
 11 files changed, 643 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

-- 
2.17.1

[PATCH v8 1/2] vhost: introduce API to get vDPA device number

[Xiaolong Ye]

It's used to get number of available registered vDPA devices.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
Acked-by: Xiao Wang <xiao.w.wang@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 lib/librte_vhost/rte_vdpa.h            | 3 +++
 lib/librte_vhost/rte_vhost_version.map | 1 +
 lib/librte_vhost/vdpa.c                | 6 ++++++
 3 files changed, 10 insertions(+)

diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
index 90465ca26..b8223e337 100644
--- a/lib/librte_vhost/rte_vdpa.h
+++ b/lib/librte_vhost/rte_vdpa.h
@@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
 struct rte_vdpa_device * __rte_experimental
 rte_vdpa_get_device(int did);
 
+/* Get current available vdpa device number */
+int __rte_experimental
+rte_vdpa_get_device_num(void);
 #endif /* _RTE_VDPA_H_ */
diff --git a/lib/librte_vhost/rte_vhost_version.map b/lib/librte_vhost/rte_vhost_version.map
index da220dd02..ae39b6e21 100644
--- a/lib/librte_vhost/rte_vhost_version.map
+++ b/lib/librte_vhost/rte_vhost_version.map
@@ -67,6 +67,7 @@ EXPERIMENTAL {
 	rte_vdpa_unregister_device;
 	rte_vdpa_find_device_id;
 	rte_vdpa_get_device;
+	rte_vdpa_get_device_num;
 	rte_vhost_driver_attach_vdpa_device;
 	rte_vhost_driver_detach_vdpa_device;
 	rte_vhost_driver_get_vdpa_device_id;
diff --git a/lib/librte_vhost/vdpa.c b/lib/librte_vhost/vdpa.c
index c82fd4370..c2c5dff1d 100644
--- a/lib/librte_vhost/vdpa.c
+++ b/lib/librte_vhost/vdpa.c
@@ -113,3 +113,9 @@ rte_vdpa_get_device(int did)
 
 	return vdpa_devices[did];
 }
+
+int
+rte_vdpa_get_device_num(void)
+{
+	return vdpa_device_num;
+}
-- 
2.17.1

Re: [PATCH v8 1/2] vhost: introduce API to get vDPA device number

[Ferruh Yigit]

On 9/28/2018 10:47 PM, Xiaolong Ye wrote:
> It's used to get number of available registered vDPA devices.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---
>  lib/librte_vhost/rte_vdpa.h            | 3 +++
>  lib/librte_vhost/rte_vhost_version.map | 1 +
>  lib/librte_vhost/vdpa.c                | 6 ++++++
>  3 files changed, 10 insertions(+)
> 
> diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
> index 90465ca26..b8223e337 100644
> --- a/lib/librte_vhost/rte_vdpa.h
> +++ b/lib/librte_vhost/rte_vdpa.h
> @@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
>  struct rte_vdpa_device * __rte_experimental
>  rte_vdpa_get_device(int did);
>  
> +/* Get current available vdpa device number */
> +int __rte_experimental
> +rte_vdpa_get_device_num(void);

Hi Xiaolong,

Is rte_vdpa_get_device_num() an public API? (It should be since it will be used
in an application in next patch.)

If so, header needs to be part of API documentation:
1- Update doc/api/doxy-api-index.md to include this header.
2- Update all APIs comments to be an doxygen comment.
3- Add doxygen comment to all data structures.

I can see rte_vdpa.h not documented at all, it is not just this API, so it can
be better to send a new patch to address above items.

Thanks,
ferruh

Re: [PATCH v8 1/2] vhost: introduce API to get vDPA device number

[Ye Xiaolong]

Hi, Ferruh,

On 10/02, Ferruh Yigit wrote:
>On 9/28/2018 10:47 PM, Xiaolong Ye wrote:
>> It's used to get number of available registered vDPA devices.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
>> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>> ---
>>  lib/librte_vhost/rte_vdpa.h            | 3 +++
>>  lib/librte_vhost/rte_vhost_version.map | 1 +
>>  lib/librte_vhost/vdpa.c                | 6 ++++++
>>  3 files changed, 10 insertions(+)
>> 
>> diff --git a/lib/librte_vhost/rte_vdpa.h b/lib/librte_vhost/rte_vdpa.h
>> index 90465ca26..b8223e337 100644
>> --- a/lib/librte_vhost/rte_vdpa.h
>> +++ b/lib/librte_vhost/rte_vdpa.h
>> @@ -84,4 +84,7 @@ rte_vdpa_find_device_id(struct rte_vdpa_dev_addr *addr);
>>  struct rte_vdpa_device * __rte_experimental
>>  rte_vdpa_get_device(int did);
>>  
>> +/* Get current available vdpa device number */
>> +int __rte_experimental
>> +rte_vdpa_get_device_num(void);
>
>Hi Xiaolong,
>
>Is rte_vdpa_get_device_num() an public API? (It should be since it will be used
>in an application in next patch.)
>
>If so, header needs to be part of API documentation:
>1- Update doc/api/doxy-api-index.md to include this header.
>2- Update all APIs comments to be an doxygen comment.
>3- Add doxygen comment to all data structures.
>
>I can see rte_vdpa.h not documented at all, it is not just this API, so it can
>be better to send a new patch to address above items.

Thanks for the reminder, I'll send a new patch for it.

Thanks,
Xiaolong
>
>Thanks,
>ferruh

[PATCH v8 2/2] examples/vdpa: introduce a new sample for vDPA

[Xiaolong Ye]

The vdpa sample application creates vhost-user sockets by using the
vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
virtio ring compatible devices to serve virtio driver directly to enable
datapath acceleration. As vDPA driver can help to set up vhost datapath,
this application doesn't need to launch dedicated worker threads for vhost
enqueue/dequeue operations.

Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
Acked-by: Xiao Wang <xiao.w.wang@intel.com>
Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
---
 MAINTAINERS                            |   2 +
 doc/guides/rel_notes/release_18_11.rst |   8 +
 doc/guides/sample_app_ug/index.rst     |   1 +
 doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
 examples/Makefile                      |   2 +-
 examples/vdpa/Makefile                 |  32 ++
 examples/vdpa/main.c                   | 453 +++++++++++++++++++++++++
 examples/vdpa/meson.build              |  16 +
 8 files changed, 633 insertions(+), 1 deletion(-)
 create mode 100644 doc/guides/sample_app_ug/vdpa.rst
 create mode 100644 examples/vdpa/Makefile
 create mode 100644 examples/vdpa/main.c
 create mode 100644 examples/vdpa/meson.build

diff --git a/MAINTAINERS b/MAINTAINERS
index 5967c1dd3..5656f18e8 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -683,6 +683,8 @@ F: doc/guides/sample_app_ug/vhost.rst
 F: examples/vhost_scsi/
 F: doc/guides/sample_app_ug/vhost_scsi.rst
 F: examples/vhost_crypto/
+F: examples/vdpa/
+F: doc/guides/sample_app_ug/vdpa.rst
 
 Vhost PMD
 M: Maxime Coquelin <maxime.coquelin@redhat.com>
diff --git a/doc/guides/rel_notes/release_18_11.rst b/doc/guides/rel_notes/release_18_11.rst
index bc9b74ec4..dd53a9ecf 100644
--- a/doc/guides/rel_notes/release_18_11.rst
+++ b/doc/guides/rel_notes/release_18_11.rst
@@ -67,6 +67,14 @@ New Features
   SR-IOV option in Hyper-V and Azure. This is an alternative to the previous
   vdev_netvsc, tap, and failsafe drivers combination.
 
+* **Add a new sample for vDPA**
+
+  The vdpa sample application creates vhost-user sockets by using the
+  vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+  virtio ring compatible devices to serve virtio driver directly to enable
+  datapath acceleration. As vDPA driver can help to set up vhost datapath,
+  this application doesn't need to launch dedicated worker threads for vhost
+  enqueue/dequeue operations.
 
 API Changes
 -----------
diff --git a/doc/guides/sample_app_ug/index.rst b/doc/guides/sample_app_ug/index.rst
index 5bedf4f6f..74b12af85 100644
--- a/doc/guides/sample_app_ug/index.rst
+++ b/doc/guides/sample_app_ug/index.rst
@@ -45,6 +45,7 @@ Sample Applications User Guides
     vhost
     vhost_scsi
     vhost_crypto
+    vdpa
     netmap_compatibility
     ip_pipeline
     test_pipeline
diff --git a/doc/guides/sample_app_ug/vdpa.rst b/doc/guides/sample_app_ug/vdpa.rst
new file mode 100644
index 000000000..745f196ca
--- /dev/null
+++ b/doc/guides/sample_app_ug/vdpa.rst
@@ -0,0 +1,120 @@
+..  SPDX-License-Identifier: BSD-3-Clause
+    Copyright(c) 2018 Intel Corporation.
+
+Vdpa Sample Application
+=======================
+
+The vdpa sample application creates vhost-user sockets by using the
+vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
+virtio ring compatible devices to serve virtio driver directly to enable
+datapath acceleration. As vDPA driver can help to set up vhost datapath,
+this application doesn't need to launch dedicated worker threads for vhost
+enqueue/dequeue operations.
+
+Testing steps
+-------------
+
+This section shows the steps of how to start VMs with vDPA vhost-user
+backend and verify network connection & live migration.
+
+Build
+~~~~~
+
+To compile the sample application see :doc:`compiling`.
+
+The application is located in the ``vdpa`` sub-directory.
+
+Start the vdpa example
+~~~~~~~~~~~~~~~~~~~~~~
+
+.. code-block:: console
+
+        ./vdpa [EAL options]  -- [--client] [--interactive|-i] or [--iface SOCKET_PATH]
+
+where
+
+* --client means running vdpa app in client mode, in the client mode, QEMU needs
+  to run as the server mode and take charge of socket file creation.
+* --iface specifies the path prefix of the UNIX domain socket file, e.g.
+  /tmp/vhost-user-, then the socket files will be named as /tmp/vhost-user-<n>
+  (n starts from 0).
+* --interactive means run the vdpa sample in interactive mode, currently 4
+  internal cmds are supported:
+
+  1. help: show help message
+  2. list: list all available vdpa devices
+  3. create: create a new vdpa port with socket file and vdpa device address
+  4. quit: unregister vhost driver and exit the application
+
+Take IFCVF driver for example:
+
+.. code-block:: console
+
+        ./vdpa -c 0x2 -n 4 --socket-mem 1024,1024 \
+                -w 0000:06:00.3,vdpa=1 -w 0000:06:00.4,vdpa=1 \
+                -- --interactive
+
+.. note::
+    Here 0000:06:00.3 and 0000:06:00.4 refer to virtio ring compatible devices,
+    and we need to bind vfio-pci to them before running vdpa sample.
+
+    * modprobe vfio-pci
+    * ./usertools/dpdk-devbind.py -b vfio-pci 06:00.3 06:00.4
+
+Then we can create 2 vdpa ports in interactive cmdline.
+
+.. code-block:: console
+
+        vdpa> list
+        device id       device address  queue num       supported features
+        0               0000:06:00.3    1               0x14c238020
+        1               0000:06:00.4    1               0x14c238020
+        2               0000:06:00.5    1               0x14c238020
+
+        vdpa> create /tmp/vdpa-socket0 0000:06:00.3
+        vdpa> create /tmp/vdpa-socket1 0000:06:00.4
+
+.. _vdpa_app_run_vm:
+
+Start the VMs
+~~~~~~~~~~~~~
+
+.. code-block:: console
+
+       qemu-system-x86_64 -cpu host -enable-kvm \
+       <snip>
+       -mem-prealloc \
+       -chardev socket,id=char0,path=<socket_file created in above steps> \
+       -netdev type=vhost-user,id=vdpa,chardev=char0 \
+       -device virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
+
+After the VMs launches, we can login the VMs and configure the ip, verify the
+network connection via ping or netperf.
+
+.. note::
+    Suggest to use QEMU 3.0.0 which extends vhost-user for vDPA.
+
+Live Migration
+~~~~~~~~~~~~~~
+vDPA supports cross-backend live migration, user can migrate SW vhost backend
+VM to vDPA backend VM and vice versa. Here are the detailed steps. Assume A is
+the source host with SW vhost VM and B is the destination host with vDPA.
+
+1. Start vdpa sample and launch a VM with exact same parameters as the VM on A,
+   in migration-listen mode:
+
+.. code-block:: console
+
+        B: <qemu-command-line> -incoming tcp:0:4444 (or other PORT))
+
+2. Start the migration (on source host):
+
+.. code-block:: console
+
+        A: (qemu) migrate -d tcp:<B ip>:4444 (or other PORT)
+
+3. Check the status (on source host):
+
+.. code-block:: console
+
+        A: (qemu) info migrate
diff --git a/examples/Makefile b/examples/Makefile
index 481720cb6..356fcb1cd 100644
--- a/examples/Makefile
+++ b/examples/Makefile
@@ -65,7 +65,7 @@ ifeq ($(CONFIG_RTE_LIBRTE_HASH),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += tep_termination
 endif
 DIRS-$(CONFIG_RTE_LIBRTE_TIMER) += timer
-DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi
+DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost vhost_scsi vdpa
 ifeq ($(CONFIG_RTE_LIBRTE_CRYPTODEV),y)
 DIRS-$(CONFIG_RTE_LIBRTE_VHOST) += vhost_crypto
 endif
diff --git a/examples/vdpa/Makefile b/examples/vdpa/Makefile
new file mode 100644
index 000000000..42672a2bc
--- /dev/null
+++ b/examples/vdpa/Makefile
@@ -0,0 +1,32 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+ifeq ($(RTE_SDK),)
+$(error "Please define RTE_SDK environment variable")
+endif
+
+# Default target, can be overridden by command line or environment
+RTE_TARGET ?= x86_64-native-linuxapp-gcc
+
+include $(RTE_SDK)/mk/rte.vars.mk
+
+ifneq ($(CONFIG_RTE_EXEC_ENV),"linuxapp")
+$(info This application can only operate in a linuxapp environment, \
+please change the definition of the RTE_TARGET environment variable)
+all:
+else
+
+# binary name
+APP = vdpa
+
+# all source are stored in SRCS-y
+SRCS-y := main.c
+
+CFLAGS += -O2 -D_FILE_OFFSET_BITS=64
+CFLAGS += $(WERROR_FLAGS)
+CFLAGS += -D_GNU_SOURCE
+CFLAGS += -DALLOW_EXPERIMENTAL_API
+
+include $(RTE_SDK)/mk/rte.extapp.mk
+
+endif
diff --git a/examples/vdpa/main.c b/examples/vdpa/main.c
new file mode 100644
index 000000000..f96fa88b7
--- /dev/null
+++ b/examples/vdpa/main.c
@@ -0,0 +1,453 @@
+/* SPDX-License-Identifier: BSD-3-Clause
+ * Copyright(c) 2018 Intel Corporation
+ */
+
+#include <getopt.h>
+#include <signal.h>
+#include <stdint.h>
+#include <string.h>
+#include <unistd.h>
+
+#include <rte_ethdev.h>
+#include <rte_malloc.h>
+#include <rte_vhost.h>
+#include <rte_vdpa.h>
+#include <rte_pci.h>
+
+#include <cmdline_parse.h>
+#include <cmdline_socket.h>
+#include <cmdline_parse_string.h>
+#include <cmdline.h>
+
+#define MAX_PATH_LEN 128
+#define MAX_VDPA_SAMPLE_PORTS 1024
+#define RTE_LOGTYPE_VDPA RTE_LOGTYPE_USER1
+
+struct vdpa_port {
+	char ifname[MAX_PATH_LEN];
+	int did;
+	int vid;
+	uint64_t flags;
+};
+
+static struct vdpa_port vports[MAX_VDPA_SAMPLE_PORTS];
+
+static char iface[MAX_PATH_LEN];
+static int dev_total;
+static int devcnt;
+static int interactive;
+static int client_mode;
+
+/* display usage */
+static void
+vdpa_usage(const char *prgname)
+{
+	printf("Usage: %s [EAL options] -- "
+				 "	--interactive|-i: run in interactive mode.\n"
+				 "	--iface <path>: specify the path prefix of the socket files, e.g. /tmp/vhost-user-.\n"
+				 "	--client: register a vhost-user socket as client mode.\n",
+				 prgname);
+}
+
+static int
+parse_args(int argc, char **argv)
+{
+	static const char *short_option = "i";
+	static struct option long_option[] = {
+		{"iface", required_argument, NULL, 0},
+		{"interactive", no_argument, &interactive, 1},
+		{"client", no_argument, &client_mode, 1},
+		{NULL, 0, 0, 0},
+	};
+	int opt, idx;
+	char *prgname = argv[0];
+
+	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
+			!= EOF) {
+		switch (opt) {
+		case 'i':
+			printf("Interactive-mode selected\n");
+			interactive = 1;
+			break;
+		/* long options */
+		case 0:
+			if (strncmp(long_option[idx].name, "iface",
+						MAX_PATH_LEN) == 0) {
+				strncpy(iface, optarg, MAX_PATH_LEN);
+				printf("iface %s\n", iface);
+			}
+			if (!strcmp(long_option[idx].name, "interactive")) {
+				printf("Interactive-mode selected\n");
+				interactive = 1;
+			}
+			break;
+
+		default:
+			vdpa_usage(prgname);
+			return -1;
+		}
+	}
+
+	if (iface[0] == '\0' && interactive == 0) {
+		vdpa_usage(prgname);
+		return -1;
+	}
+
+	return 0;
+}
+
+static int
+new_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strncmp(ifname, vports[i].ifname, MAX_PATH_LEN) == 0) {
+			printf("\nnew port %s, did: %d\n",
+					ifname, vports[i].did);
+			vports[i].vid = vid;
+			break;
+		}
+	}
+
+	if (i >= MAX_VDPA_SAMPLE_PORTS)
+		return -1;
+
+	return 0;
+}
+
+static void
+destroy_device(int vid)
+{
+	char ifname[MAX_PATH_LEN];
+	int i;
+
+	rte_vhost_get_ifname(vid, ifname, sizeof(ifname));
+	for (i = 0; i < MAX_VDPA_SAMPLE_PORTS; i++) {
+		if (strcmp(ifname, vports[i].ifname) == 0) {
+			printf("\ndestroy port %s, did: %d\n",
+					ifname, vports[i].did);
+			break;
+		}
+	}
+}
+
+static const struct vhost_device_ops vdpa_sample_devops = {
+	.new_device = new_device,
+	.destroy_device = destroy_device,
+};
+
+static int
+start_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+	int did = vport->did;
+
+	if (client_mode)
+		vport->flags |= RTE_VHOST_USER_CLIENT;
+
+	if (access(socket_path, F_OK) != -1 && !client_mode) {
+		RTE_LOG(ERR, VDPA,
+			"%s exists, please remove it or specify another file and try again.\n",
+			socket_path);
+		return -1;
+	}
+	ret = rte_vhost_driver_register(socket_path, vport->flags);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_callback_register(socket_path,
+			&vdpa_sample_devops);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"register driver ops failed: %s\n",
+			socket_path);
+
+	ret = rte_vhost_driver_attach_vdpa_device(socket_path, did);
+	if (ret != 0)
+		rte_exit(EXIT_FAILURE,
+			"attach vdpa device failed: %s\n",
+			socket_path);
+
+	if (rte_vhost_driver_start(socket_path) < 0)
+		rte_exit(EXIT_FAILURE,
+			"start vhost driver failed: %s\n",
+			socket_path);
+	return 0;
+}
+
+static void
+close_vdpa(struct vdpa_port *vport)
+{
+	int ret;
+	char *socket_path = vport->ifname;
+
+	ret = rte_vhost_driver_detach_vdpa_device(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"detach vdpa device failed: %s\n",
+				socket_path);
+
+	ret = rte_vhost_driver_unregister(socket_path);
+	if (ret != 0)
+		RTE_LOG(ERR, VDPA,
+				"Fail to unregister vhost driver for %s.\n",
+				socket_path);
+}
+
+static void
+vdpa_sample_quit(void)
+{
+	int i;
+	for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total); i++) {
+		if (vports[i].ifname[0] != '\0')
+			close_vdpa(&vports[i]);
+	}
+}
+
+static void
+signal_handler(int signum)
+{
+	if (signum == SIGINT || signum == SIGTERM) {
+		printf("\nSignal %d received, preparing to exit...\n", signum);
+		vdpa_sample_quit();
+		exit(0);
+	}
+}
+
+/* interactive cmds */
+
+/* *** Help command with introduction. *** */
+struct cmd_help_result {
+	cmdline_fixed_string_t help;
+};
+
+static void cmd_help_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	cmdline_printf(
+		cl,
+		"\n"
+		"The following commands are currently available:\n\n"
+		"Control:\n"
+		"    help                                      : Show interactive instructions.\n"
+		"    list                                      : list all available vdpa devices.\n"
+		"    create <socket file> <vdev addr>          : create a new vdpa port.\n"
+		"    quit                                      : exit vdpa sample app.\n"
+	);
+}
+
+cmdline_parse_token_string_t cmd_help_help =
+	TOKEN_STRING_INITIALIZER(struct cmd_help_result, help, "help");
+
+cmdline_parse_inst_t cmd_help = {
+	.f = cmd_help_parsed,
+	.data = NULL,
+	.help_str = "show help",
+	.tokens = {
+		(void *)&cmd_help_help,
+		NULL,
+	},
+};
+
+/* *** List all available vdpa devices *** */
+struct cmd_list_result {
+	cmdline_fixed_string_t action;
+};
+
+static void cmd_list_vdpa_devices_parsed(
+		__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	uint32_t queue_num;
+	uint64_t features;
+	struct rte_vdpa_device *vdev;
+	struct rte_pci_addr addr;
+
+	cmdline_printf(cl, "device id\tdevice address\tqueue num\tsupported features\n");
+	for (did = 0; did < dev_total; did++) {
+		vdev = rte_vdpa_get_device(did);
+		if (!vdev)
+			continue;
+		if (vdev->ops->get_queue_num(did, &queue_num) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa queue number "
+				"for device id %d.\n", did);
+			continue;
+		}
+		if (vdev->ops->get_features(did, &features) < 0) {
+			RTE_LOG(ERR, VDPA,
+				"failed to get vdpa features "
+				"for device id %d.\n", did);
+			continue;
+		}
+		addr = vdev->addr.pci_addr;
+		cmdline_printf(cl,
+			"%d\t\t" PCI_PRI_FMT "\t%" PRIu32 "\t\t0x%" PRIx64 "\n",
+			did, addr.domain, addr.bus, addr.devid,
+			addr.function, queue_num, features);
+	}
+}
+
+cmdline_parse_token_string_t cmd_action_list =
+	TOKEN_STRING_INITIALIZER(struct cmd_list_result, action, "list");
+
+cmdline_parse_inst_t cmd_list_vdpa_devices = {
+	.f = cmd_list_vdpa_devices_parsed,
+	.data = NULL,
+	.help_str = "list all available vdpa devices",
+	.tokens = {
+		(void *)&cmd_action_list,
+		NULL,
+	},
+};
+
+/* *** Create new vdpa port *** */
+struct cmd_create_result {
+	cmdline_fixed_string_t action;
+	cmdline_fixed_string_t socket_path;
+	cmdline_fixed_string_t bdf;
+};
+
+static void cmd_create_vdpa_port_parsed(void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	int did;
+	struct cmd_create_result *res = parsed_result;
+	struct rte_vdpa_dev_addr addr;
+
+	strncpy(vports[devcnt].ifname, res->socket_path, MAX_PATH_LEN);
+	if (rte_pci_addr_parse(res->bdf, &addr.pci_addr) != 0) {
+		cmdline_printf(cl, "Unable to parse the given bdf.\n");
+		return;
+	}
+	addr.type = PCI_ADDR;
+	did = rte_vdpa_find_device_id(&addr);
+	if (did < 0) {
+		cmdline_printf(cl, "Unable to find vdpa devide id.\n");
+		return;
+	}
+
+	vports[devcnt].did = did;
+
+	if (start_vdpa(&vports[devcnt]) == 0)
+		devcnt++;
+}
+
+cmdline_parse_token_string_t cmd_action_create =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, action, "create");
+cmdline_parse_token_string_t cmd_socket_path =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, socket_path, NULL);
+cmdline_parse_token_string_t cmd_bdf =
+	TOKEN_STRING_INITIALIZER(struct cmd_create_result, bdf, NULL);
+
+cmdline_parse_inst_t cmd_create_vdpa_port = {
+	.f = cmd_create_vdpa_port_parsed,
+	.data = NULL,
+	.help_str = "create a new vdpa port",
+	.tokens = {
+		(void *)&cmd_action_create,
+		(void *)&cmd_socket_path,
+		(void *)&cmd_bdf,
+		NULL,
+	},
+};
+
+/* *** QUIT *** */
+struct cmd_quit_result {
+	cmdline_fixed_string_t quit;
+};
+
+static void cmd_quit_parsed(__attribute__((unused)) void *parsed_result,
+		struct cmdline *cl,
+		__attribute__((unused)) void *data)
+{
+	vdpa_sample_quit();
+	cmdline_quit(cl);
+}
+
+cmdline_parse_token_string_t cmd_quit_quit =
+	TOKEN_STRING_INITIALIZER(struct cmd_quit_result, quit, "quit");
+
+cmdline_parse_inst_t cmd_quit = {
+	.f = cmd_quit_parsed,
+	.data = NULL,
+	.help_str = "quit: exit application",
+	.tokens = {
+		(void *)&cmd_quit_quit,
+		NULL,
+	},
+};
+cmdline_parse_ctx_t main_ctx[] = {
+	(cmdline_parse_inst_t *)&cmd_help,
+	(cmdline_parse_inst_t *)&cmd_list_vdpa_devices,
+	(cmdline_parse_inst_t *)&cmd_create_vdpa_port,
+	(cmdline_parse_inst_t *)&cmd_quit,
+	NULL,
+};
+
+int
+main(int argc, char *argv[])
+{
+	char ch;
+	int i;
+	int ret;
+	struct cmdline *cl;
+
+	ret = rte_eal_init(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "eal init failed\n");
+	argc -= ret;
+	argv += ret;
+
+	dev_total = rte_vdpa_get_device_num();
+	if (dev_total <= 0)
+		rte_exit(EXIT_FAILURE, "No available vdpa device found\n");
+
+	signal(SIGINT, signal_handler);
+	signal(SIGTERM, signal_handler);
+
+	ret = parse_args(argc, argv);
+	if (ret < 0)
+		rte_exit(EXIT_FAILURE, "invalid argument\n");
+
+	if (interactive == 1) {
+		cl = cmdline_stdin_new(main_ctx, "vdpa> ");
+		if (cl == NULL)
+			rte_panic("Cannot create cmdline instance\n");
+		cmdline_interact(cl);
+		cmdline_stdin_exit(cl);
+	} else {
+		for (i = 0; i < RTE_MIN(MAX_VDPA_SAMPLE_PORTS, dev_total);
+				i++) {
+			vports[i].did = i;
+			snprintf(vports[i].ifname, MAX_PATH_LEN, "%s%d",
+					iface, i);
+
+			start_vdpa(&vports[i]);
+		}
+
+		printf("enter \'q\' to quit\n");
+		while (scanf("%c", &ch)) {
+			if (ch == 'q')
+				break;
+			while (ch != '\n') {
+				if (scanf("%c", &ch))
+					printf("%c", ch);
+			}
+			printf("enter \'q\' to quit\n");
+		}
+		vdpa_sample_quit();
+	}
+
+	return 0;
+}
diff --git a/examples/vdpa/meson.build b/examples/vdpa/meson.build
new file mode 100644
index 000000000..2e38a069f
--- /dev/null
+++ b/examples/vdpa/meson.build
@@ -0,0 +1,16 @@
+# SPDX-License-Identifier: BSD-3-Clause
+# Copyright(c) 2018 Intel Corporation
+
+# meson file, for building this example as part of a main DPDK build.
+#
+# To build this example as a standalone application with an already-installed
+# DPDK instance, use 'make'
+
+if host_machine.system() != 'linux'
+	build = false
+endif
+deps += 'vhost'
+allow_experimental_apis = true
+sources = files(
+	'main.c'
+)
\ No newline at end of file
-- 
2.17.1

Re: [PATCH v8 2/2] examples/vdpa: introduce a new sample for vDPA

[Ferruh Yigit]

On 9/28/2018 10:47 PM, Xiaolong Ye wrote:
> The vdpa sample application creates vhost-user sockets by using the
> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
> virtio ring compatible devices to serve virtio driver directly to enable
> datapath acceleration. As vDPA driver can help to set up vhost datapath,
> this application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>

<...>

> +static int
> +parse_args(int argc, char **argv)
> +{
> +	static const char *short_option = "i";
> +	static struct option long_option[] = {
> +		{"iface", required_argument, NULL, 0},
> +		{"interactive", no_argument, &interactive, 1},
> +		{"client", no_argument, &client_mode, 1},
> +		{NULL, 0, 0, 0},
> +	};
> +	int opt, idx;
> +	char *prgname = argv[0];
> +
> +	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
> +			!= EOF) {
> +		switch (opt) {
> +		case 'i':
> +			printf("Interactive-mode selected\n");
> +			interactive = 1;
> +			break;
> +		/* long options */
> +		case 0:
> +			if (strncmp(long_option[idx].name, "iface",
> +						MAX_PATH_LEN) == 0) {
> +				strncpy(iface, optarg, MAX_PATH_LEN);

Giving compiler warning [1], rte_strscpy() is safer to use.

[1]
In function ‘parse_args’,



    inlined from ‘main’ at .../examples/vdpa/main.c:419:8:



.../examples/vdpa/main.c:76:5: error: ‘strncpy’ specified bound 128 equals
destination size [-Werror=stringop-truncation]


     strncpy(iface, optarg, MAX_PATH_LEN);



     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Re: [PATCH v8 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

Hi, Ferruh,

On 10/02, Ferruh Yigit wrote:
>On 9/28/2018 10:47 PM, Xiaolong Ye wrote:
>> The vdpa sample application creates vhost-user sockets by using the
>> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
>> virtio ring compatible devices to serve virtio driver directly to enable
>> datapath acceleration. As vDPA driver can help to set up vhost datapath,
>> this application doesn't need to launch dedicated worker threads for vhost
>> enqueue/dequeue operations.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
>> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>
><...>
>
>> +static int
>> +parse_args(int argc, char **argv)
>> +{
>> +	static const char *short_option = "i";
>> +	static struct option long_option[] = {
>> +		{"iface", required_argument, NULL, 0},
>> +		{"interactive", no_argument, &interactive, 1},
>> +		{"client", no_argument, &client_mode, 1},
>> +		{NULL, 0, 0, 0},
>> +	};
>> +	int opt, idx;
>> +	char *prgname = argv[0];
>> +
>> +	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
>> +			!= EOF) {
>> +		switch (opt) {
>> +		case 'i':
>> +			printf("Interactive-mode selected\n");
>> +			interactive = 1;
>> +			break;
>> +		/* long options */
>> +		case 0:
>> +			if (strncmp(long_option[idx].name, "iface",
>> +						MAX_PATH_LEN) == 0) {
>> +				strncpy(iface, optarg, MAX_PATH_LEN);
>
>Giving compiler warning [1], rte_strscpy() is safer to use.
>
>[1]
>In function ‘parse_args’,
>
>
>
>    inlined from ‘main’ at .../examples/vdpa/main.c:419:8:
>
>
>
>.../examples/vdpa/main.c:76:5: error: ‘strncpy’ specified bound 128 equals
>destination size [-Werror=stringop-truncation]
>
>
>     strncpy(iface, optarg, MAX_PATH_LEN);
>
>
>
>     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Thanks for the finding, by the way, how do you get this warning? I can't see
it in my local build, what compiler you used?

Thanks,
Xiaolong

Re: [PATCH v8 2/2] examples/vdpa: introduce a new sample for vDPA

[Ferruh Yigit]

On 10/3/2018 7:04 AM, Ye Xiaolong wrote:
> Hi, Ferruh,
> 
> On 10/02, Ferruh Yigit wrote:
>> On 9/28/2018 10:47 PM, Xiaolong Ye wrote:
>>> The vdpa sample application creates vhost-user sockets by using the
>>> vDPA backend. vDPA stands for vhost Data Path Acceleration which utilizes
>>> virtio ring compatible devices to serve virtio driver directly to enable
>>> datapath acceleration. As vDPA driver can help to set up vhost datapath,
>>> this application doesn't need to launch dedicated worker threads for vhost
>>> enqueue/dequeue operations.
>>>
>>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>>> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
>>> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>>
>> <...>
>>
>>> +static int
>>> +parse_args(int argc, char **argv)
>>> +{
>>> +	static const char *short_option = "i";
>>> +	static struct option long_option[] = {
>>> +		{"iface", required_argument, NULL, 0},
>>> +		{"interactive", no_argument, &interactive, 1},
>>> +		{"client", no_argument, &client_mode, 1},
>>> +		{NULL, 0, 0, 0},
>>> +	};
>>> +	int opt, idx;
>>> +	char *prgname = argv[0];
>>> +
>>> +	while ((opt = getopt_long(argc, argv, short_option, long_option, &idx))
>>> +			!= EOF) {
>>> +		switch (opt) {
>>> +		case 'i':
>>> +			printf("Interactive-mode selected\n");
>>> +			interactive = 1;
>>> +			break;
>>> +		/* long options */
>>> +		case 0:
>>> +			if (strncmp(long_option[idx].name, "iface",
>>> +						MAX_PATH_LEN) == 0) {
>>> +				strncpy(iface, optarg, MAX_PATH_LEN);
>>
>> Giving compiler warning [1], rte_strscpy() is safer to use.
>>
>> [1]
>> In function ‘parse_args’,
>>
>>
>>
>>    inlined from ‘main’ at .../examples/vdpa/main.c:419:8:
>>
>>
>>
>> .../examples/vdpa/main.c:76:5: error: ‘strncpy’ specified bound 128 equals
>> destination size [-Werror=stringop-truncation]
>>
>>
>>     strncpy(iface, optarg, MAX_PATH_LEN);
>>
>>
>>
>>     ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> 
> Thanks for the finding, by the way, how do you get this warning? I can't see
> it in my local build, what compiler you used?

gcc,

$ gcc --version
gcc (GCC) 8.1.1 20180712 (Red Hat 8.1.1-5)

Re: [PATCH v8 2/2] examples/vdpa: introduce a new sample for vDPA

[Shahaf Shuler]

Hi, 

Going back to this merged commit, 

Saturday, September 29, 2018 12:48 AM, Xiaolong Ye:
> Subject: [dpdk-dev] [PATCH v8 2/2] examples/vdpa: introduce a new sample
> for vDPA
> 
> The vdpa sample application creates vhost-user sockets by using the vDPA
> backend. vDPA stands for vhost Data Path Acceleration which utilizes virtio
> ring compatible devices to serve virtio driver directly to enable datapath
> acceleration. As vDPA driver can help to set up vhost datapath, this
> application doesn't need to launch dedicated worker threads for vhost
> enqueue/dequeue operations.
> 
> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> ---

[...]

> +
> +Start the VMs
> +~~~~~~~~~~~~~
> +
> +.. code-block:: console
> +
> +       qemu-system-x86_64 -cpu host -enable-kvm \
> +       <snip>
> +       -mem-prealloc \
> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> +       -device
> + virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \


It doesn't seems like qemu has the id=vdpa and netdev=vdpa options. 
Grepping vdpa on qemu returns empty. 

Am using QEMU v3.1.0-513-gfb177a0. Commit head is fb177a036f2e ("MAINTAINERS: Mark RISC-V as Supported").

Did I missed anything?

Re: [PATCH v8 2/2] examples/vdpa: introduce a new sample for vDPA

[Ye Xiaolong]

Hi,

On 01/17, Shahaf Shuler wrote:
>Hi, 
>
>Going back to this merged commit, 
>
>Saturday, September 29, 2018 12:48 AM, Xiaolong Ye:
>> Subject: [dpdk-dev] [PATCH v8 2/2] examples/vdpa: introduce a new sample
>> for vDPA
>> 
>> The vdpa sample application creates vhost-user sockets by using the vDPA
>> backend. vDPA stands for vhost Data Path Acceleration which utilizes virtio
>> ring compatible devices to serve virtio driver directly to enable datapath
>> acceleration. As vDPA driver can help to set up vhost datapath, this
>> application doesn't need to launch dedicated worker threads for vhost
>> enqueue/dequeue operations.
>> 
>> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
>> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
>> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>> ---
>
>[...]
>
>> +
>> +Start the VMs
>> +~~~~~~~~~~~~~
>> +
>> +.. code-block:: console
>> +
>> +       qemu-system-x86_64 -cpu host -enable-kvm \
>> +       <snip>
>> +       -mem-prealloc \
>> +       -chardev socket,id=char0,path=<socket_file created in above steps> \
>> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
>> +       -device
>> + virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
>
>
>It doesn't seems like qemu has the id=vdpa and netdev=vdpa options. 
>Grepping vdpa on qemu returns empty. 
>
>Am using QEMU v3.1.0-513-gfb177a0. Commit head is fb177a036f2e ("MAINTAINERS: Mark RISC-V as Supported").
>
>Did I missed anything?
>

Not quite sure about your question here, the vdpa used in above qemu cmdline
is just a normal string, you can replace it with other string like 

-netdev type=vhost-user,id=mynet,chardev=char0 \
-device virtio-net-pci,netdev=mynet,...

The major support for vdpa is done in dpdk side as the vhost user backend, qemu
will use normal vhost user protocol to communicate with it.

Thanks,
Xiaolong

Re: [PATCH v8 2/2] examples/vdpa: introduce a new sample for vDPA

[Shahaf Shuler]

Thursday, January 17, 2019 5:13 PM, Ye Xiaolong:
> Subject: Re: [dpdk-dev] [PATCH v8 2/2] examples/vdpa: introduce a new
> sample for vDPA
> 
> Hi,
> 
> On 01/17, Shahaf Shuler wrote:
> >Hi,
> >
> >Going back to this merged commit,
> >
> >Saturday, September 29, 2018 12:48 AM, Xiaolong Ye:
> >> Subject: [dpdk-dev] [PATCH v8 2/2] examples/vdpa: introduce a new
> >> sample for vDPA
> >>
> >> The vdpa sample application creates vhost-user sockets by using the
> >> vDPA backend. vDPA stands for vhost Data Path Acceleration which
> >> utilizes virtio ring compatible devices to serve virtio driver
> >> directly to enable datapath acceleration. As vDPA driver can help to
> >> set up vhost datapath, this application doesn't need to launch
> >> dedicated worker threads for vhost enqueue/dequeue operations.
> >>
> >> Signed-off-by: Xiaolong Ye <xiaolong.ye@intel.com>
> >> Acked-by: Xiao Wang <xiao.w.wang@intel.com>
> >> Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> >> ---
> >
> >[...]
> >
> >> +
> >> +Start the VMs
> >> +~~~~~~~~~~~~~
> >> +
> >> +.. code-block:: console
> >> +
> >> +       qemu-system-x86_64 -cpu host -enable-kvm \
> >> +       <snip>
> >> +       -mem-prealloc \
> >> +       -chardev socket,id=char0,path=<socket_file created in above steps>
> \
> >> +       -netdev type=vhost-user,id=vdpa,chardev=char0 \
> >> +       -device
> >> + virtio-net-pci,netdev=vdpa,mac=00:aa:bb:cc:dd:ee,page-per-vq=on \
> >
> >
> >It doesn't seems like qemu has the id=vdpa and netdev=vdpa options.
> >Grepping vdpa on qemu returns empty.
> >
> >Am using QEMU v3.1.0-513-gfb177a0. Commit head is fb177a036f2e
> ("MAINTAINERS: Mark RISC-V as Supported").
> >
> >Did I missed anything?
> >
> 
> Not quite sure about your question here, the vdpa used in above qemu
> cmdline is just a normal string, you can replace it with other string like
> 
> -netdev type=vhost-user,id=mynet,chardev=char0 \ -device virtio-net-
> pci,netdev=mynet,...
> 
> The major support for vdpa is done in dpdk side as the vhost user backend,
> qemu will use normal vhost user protocol to communicate with it.

Thanks. I mis-interpreted the option fields.

> 
> Thanks,
> Xiaolong

Re: [PATCH v8 0/2] introduce vdpa sample

[Maxime Coquelin]

On 09/28/2018 11:47 PM, Xiaolong Ye wrote:
> Hi,
> 
> This patchset introduces vdpa sample to demonstrate the vDPA use case.
> 
> v8 changes:
> * move body of data_init to the main function
> * adjust Reviewed-by/Acked-by tag postion
> 
> v7 changes:
> * make vports static
> * avoid unnecessary static variable initialization
> 
> v6 changes:
> * improve the document according to Xiao's comments
> * fix a typo, PRIu64 -> PRIx64
> 
> v5 changes:
> * improve print format and correct from "PRIu64" to "PRIx64"
> * use "-c 0x2" to better demonstrate app doesn't need to launch dedicated
>    worker threads for vhost enqueue/dequeue operations in vdpa.rst
> 
> v4 changes:
> * add client mode support
> * improve the format to list the vDPA device info and improve the vdpa.rst
>    accordingly
> * remove some useless comments
> * add introduction in 18.11 release note.
> 
> 
> v3 changes:
> * list cmd would show queue number and supported features of vdpa devices.
> * address Xiao's review comments
> 
> v2 changes:
> 
> * fix a compilation error reported by Rosen
> * improve create cmd in interactive mode and add two new cmds: list, quit
> * add application documentation
> 
> 
> Xiaolong Ye (2):
>    vhost: introduce API to get vDPA device number
>    examples/vdpa: introduce a new sample for vDPA
> 
>   MAINTAINERS                            |   2 +
>   doc/guides/rel_notes/release_18_11.rst |   8 +
>   doc/guides/sample_app_ug/index.rst     |   1 +
>   doc/guides/sample_app_ug/vdpa.rst      | 120 +++++++
>   examples/Makefile                      |   2 +-
>   examples/vdpa/Makefile                 |  32 ++
>   examples/vdpa/main.c                   | 453 +++++++++++++++++++++++++
>   examples/vdpa/meson.build              |  16 +
>   lib/librte_vhost/rte_vdpa.h            |   3 +
>   lib/librte_vhost/rte_vhost_version.map |   1 +
>   lib/librte_vhost/vdpa.c                |   6 +
>   11 files changed, 643 insertions(+), 1 deletion(-)
>   create mode 100644 doc/guides/sample_app_ug/vdpa.rst
>   create mode 100644 examples/vdpa/Makefile
>   create mode 100644 examples/vdpa/main.c
>   create mode 100644 examples/vdpa/meson.build
> 

Applied to dpdk-next-virtio/master.

Thanks,
Maxime