[PATCH V4 0/3] virtio-pci: introduce modern device module

[PATCH V4 0/3] virtio-pci: introduce modern device module

[Jason Wang]

Hi all:

This series tries to implement a vDPA driver for virtio-pci device
which will bridge between vDPA bus and virtio-pci device.

This could be used for future feature prototyping and testing.

Please review

Changes since V4:
- include the patch to let VIRTIO_PCI_LIB to be auto selected
- style fixes in the Kconfig for vp-vdpa driver
- fix the err value returned during vp-vdpa driver probing

Changes since V3:
- rebase to vhost.git

Changes since V2:

- tweak config prompt
- switch from 'cb' to 'config_cb' for vp_vdpa config interrupt
- use a macro for vp_vdpa msix name length

Changes since V1:

- don't try to use devres for virtio-pci core
- tweak the commit log
- split the patches furtherly to ease the reviewing

Changes since RFC:

- Split common codes from virito-pci and share it with vDPA driver
- Use dynamic id in order to be less confusing with virtio-pci driver
- No feature whitelist, supporting any features (mq, config etc)

Jason Wang (3):
  virtio: don't prompt CONFIG_VIRTIO_PCI_MODERN
  vdpa: set the virtqueue num during register
  vdpa: introduce virtio pci driver

 drivers/vdpa/Kconfig                 |   7 +
 drivers/vdpa/Makefile                |   1 +
 drivers/vdpa/ifcvf/ifcvf_main.c      |   5 +-
 drivers/vdpa/mlx5/net/mlx5_vnet.c    |   4 +-
 drivers/vdpa/vdpa.c                  |  18 +-
 drivers/vdpa/vdpa_sim/vdpa_sim.c     |   2 +-
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |   2 +-
 drivers/vdpa/virtio_pci/Makefile     |   2 +
 drivers/vdpa/virtio_pci/vp_vdpa.c    | 458 +++++++++++++++++++++++++++
 drivers/virtio/Kconfig               |  11 +-
 drivers/virtio/Makefile              |   2 +-
 include/linux/vdpa.h                 |  10 +-
 12 files changed, 496 insertions(+), 26 deletions(-)
 create mode 100644 drivers/vdpa/virtio_pci/Makefile
 create mode 100644 drivers/vdpa/virtio_pci/vp_vdpa.c

-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V4 1/3] virtio: don't prompt CONFIG_VIRTIO_PCI_MODERN

[Jason Wang]

We used to prompt CONFIG_VIRTIO_PCI_MODERN to user which may bring a
lot of confusion. E.g it may break various default configs which want
virtio devices.

So this patch fixes this by hiding the prompot and documenting the
dependency. While at it, rename the module to VIRTIO_PCI_LIB.

Cc: Arnd Bergmann <arnd@arndb.de>
Cc: Anders Roxell <anders.roxell@linaro.org>
Cc: Guenter Roeck <linux@roeck-us.net>
Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
Fixes: 86b87c9d858b6 ("virtio-pci: introduce modern device module")
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/virtio/Kconfig  | 11 ++++++-----
 drivers/virtio/Makefile |  2 +-
 2 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
index 6b9b81f4b8c2..ce1b3f6ec325 100644
--- a/drivers/virtio/Kconfig
+++ b/drivers/virtio/Kconfig
@@ -12,13 +12,13 @@ config ARCH_HAS_RESTRICTED_VIRTIO_MEMORY_ACCESS
 	  This option is selected if the architecture may need to enforce
 	  VIRTIO_F_ACCESS_PLATFORM
 
-config VIRTIO_PCI_MODERN
-	tristate "Modern Virtio PCI Device"
-	depends on PCI
+config VIRTIO_PCI_LIB
+	tristate
 	help
 	  Modern PCI device implementation. This module implements the
 	  basic probe and control for devices which are based on modern
-	  PCI device with possible vendor specific extensions.
+	  PCI device with possible vendor specific extensions. Any
+	  module that selects this module must depend on PCI.
 
 menuconfig VIRTIO_MENU
 	bool "Virtio drivers"
@@ -28,7 +28,8 @@ if VIRTIO_MENU
 
 config VIRTIO_PCI
 	tristate "PCI driver for virtio devices"
-	depends on VIRTIO_PCI_MODERN
+	depends on PCI
+	select VIRTIO_PCI_LIB
 	select VIRTIO
 	help
 	  This driver provides support for virtio based paravirtual device
diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
index f097578aaa8f..699bbea0465f 100644
--- a/drivers/virtio/Makefile
+++ b/drivers/virtio/Makefile
@@ -1,6 +1,6 @@
 # SPDX-License-Identifier: GPL-2.0
 obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o
-obj-$(CONFIG_VIRTIO_PCI_MODERN) += virtio_pci_modern_dev.o
+obj-$(CONFIG_VIRTIO_PCI_LIB) += virtio_pci_modern_dev.o
 obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o
 obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o
 virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 1/3] virtio: don't prompt CONFIG_VIRTIO_PCI_MODERN

[Guenter Roeck]

On 2/22/21 10:19 PM, Jason Wang wrote:
> We used to prompt CONFIG_VIRTIO_PCI_MODERN to user which may bring a
> lot of confusion. E.g it may break various default configs which want
> virtio devices.
> 
> So this patch fixes this by hiding the prompot and documenting the
> dependency. While at it, rename the module to VIRTIO_PCI_LIB.
> 
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Anders Roxell <anders.roxell@linaro.org>
> Cc: Guenter Roeck <linux@roeck-us.net>
> Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
> Fixes: 86b87c9d858b6 ("virtio-pci: introduce modern device module")
> Signed-off-by: Jason Wang <jasowang@redhat.com>

Acked-by: Guenter Roeck <linux@roeck-us.net>

> ---
>  drivers/virtio/Kconfig  | 11 ++++++-----
>  drivers/virtio/Makefile |  2 +-
>  2 files changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
> index 6b9b81f4b8c2..ce1b3f6ec325 100644
> --- a/drivers/virtio/Kconfig
> +++ b/drivers/virtio/Kconfig
> @@ -12,13 +12,13 @@ config ARCH_HAS_RESTRICTED_VIRTIO_MEMORY_ACCESS
>  	  This option is selected if the architecture may need to enforce
>  	  VIRTIO_F_ACCESS_PLATFORM
>  
> -config VIRTIO_PCI_MODERN
> -	tristate "Modern Virtio PCI Device"
> -	depends on PCI
> +config VIRTIO_PCI_LIB
> +	tristate
>  	help
>  	  Modern PCI device implementation. This module implements the
>  	  basic probe and control for devices which are based on modern
> -	  PCI device with possible vendor specific extensions.
> +	  PCI device with possible vendor specific extensions. Any
> +	  module that selects this module must depend on PCI.

Nit: "depends on PCI" might have been useful here, since the config
system would then complain about any violations.

Thanks,
Guenter

>  
>  menuconfig VIRTIO_MENU
>  	bool "Virtio drivers"
> @@ -28,7 +28,8 @@ if VIRTIO_MENU
>  
>  config VIRTIO_PCI
>  	tristate "PCI driver for virtio devices"
> -	depends on VIRTIO_PCI_MODERN
> +	depends on PCI
> +	select VIRTIO_PCI_LIB
>  	select VIRTIO
>  	help
>  	  This driver provides support for virtio based paravirtual device
> diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
> index f097578aaa8f..699bbea0465f 100644
> --- a/drivers/virtio/Makefile
> +++ b/drivers/virtio/Makefile
> @@ -1,6 +1,6 @@
>  # SPDX-License-Identifier: GPL-2.0
>  obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o
> -obj-$(CONFIG_VIRTIO_PCI_MODERN) += virtio_pci_modern_dev.o
> +obj-$(CONFIG_VIRTIO_PCI_LIB) += virtio_pci_modern_dev.o
>  obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o
>  obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o
>  virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
> 

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 1/3] virtio: don't prompt CONFIG_VIRTIO_PCI_MODERN

[Michael S. Tsirkin]

On Tue, Feb 23, 2021 at 09:50:35AM -0600, Daniel Díaz wrote:
> Hello!
> 
> If I may add to the nit-picking...
> 
> On Tue, 23 Feb 2021 at 09:23, Guenter Roeck <linux@roeck-us.net> wrote:
> > On 2/22/21 10:19 PM, Jason Wang wrote:
> > > We used to prompt CONFIG_VIRTIO_PCI_MODERN to user which may bring a
> > > lot of confusion. E.g it may break various default configs which want
> > > virtio devices.
> > >
> > > So this patch fixes this by hiding the prompot and documenting the
> 
> *prompt
> 
> Greetings!
> 
> Daniel Díaz
> daniel.diaz@linaro.org

I have just folded this patch into the one it's fixing,
so the commit log doesn't matter. Thanks everyone!

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 1/3] virtio: don't prompt CONFIG_VIRTIO_PCI_MODERN

[Christian Borntraeger]

On 23.02.21 07:19, Jason Wang wrote:
> We used to prompt CONFIG_VIRTIO_PCI_MODERN to user which may bring a
> lot of confusion. E.g it may break various default configs which want
> virtio devices.
> 
> So this patch fixes this by hiding the prompot and documenting the
> dependency. While at it, rename the module to VIRTIO_PCI_LIB.
> 
> Cc: Arnd Bergmann <arnd@arndb.de>
> Cc: Anders Roxell <anders.roxell@linaro.org>
> Cc: Guenter Roeck <linux@roeck-us.net>
> Reported-by: Naresh Kamboju <naresh.kamboju@linaro.org>
> Fixes: 86b87c9d858b6 ("virtio-pci: introduce modern device module")
> Signed-off-by: Jason Wang <jasowang@redhat.com>

Michael,

Can we please add this NOW to next? virtio-pci is broken without it on
many defconfigs including s390. So linux-next is broken for more than 
2 weeks now and it actively breaks several parts of our CI.
I guess there are other CIs that will not run several testcases because
of this. And Naresh reported that on Feb 9th.
There IS value in CI tools on next. Not caring about regressions introduced
by a tree in next is harmful. Especially when we are close or in the
merge window. So please: either fix things quickly OR revert.

Christian



> ---
>  drivers/virtio/Kconfig  | 11 ++++++-----
>  drivers/virtio/Makefile |  2 +-
>  2 files changed, 7 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/virtio/Kconfig b/drivers/virtio/Kconfig
> index 6b9b81f4b8c2..ce1b3f6ec325 100644
> --- a/drivers/virtio/Kconfig
> +++ b/drivers/virtio/Kconfig
> @@ -12,13 +12,13 @@ config ARCH_HAS_RESTRICTED_VIRTIO_MEMORY_ACCESS
>  	  This option is selected if the architecture may need to enforce
>  	  VIRTIO_F_ACCESS_PLATFORM
>  
> -config VIRTIO_PCI_MODERN
> -	tristate "Modern Virtio PCI Device"
> -	depends on PCI
> +config VIRTIO_PCI_LIB
> +	tristate
>  	help
>  	  Modern PCI device implementation. This module implements the
>  	  basic probe and control for devices which are based on modern
> -	  PCI device with possible vendor specific extensions.
> +	  PCI device with possible vendor specific extensions. Any
> +	  module that selects this module must depend on PCI.
>  
>  menuconfig VIRTIO_MENU
>  	bool "Virtio drivers"
> @@ -28,7 +28,8 @@ if VIRTIO_MENU
>  
>  config VIRTIO_PCI
>  	tristate "PCI driver for virtio devices"
> -	depends on VIRTIO_PCI_MODERN
> +	depends on PCI
> +	select VIRTIO_PCI_LIB
>  	select VIRTIO
>  	help
>  	  This driver provides support for virtio based paravirtual device
> diff --git a/drivers/virtio/Makefile b/drivers/virtio/Makefile
> index f097578aaa8f..699bbea0465f 100644
> --- a/drivers/virtio/Makefile
> +++ b/drivers/virtio/Makefile
> @@ -1,6 +1,6 @@
>  # SPDX-License-Identifier: GPL-2.0
>  obj-$(CONFIG_VIRTIO) += virtio.o virtio_ring.o
> -obj-$(CONFIG_VIRTIO_PCI_MODERN) += virtio_pci_modern_dev.o
> +obj-$(CONFIG_VIRTIO_PCI_LIB) += virtio_pci_modern_dev.o
>  obj-$(CONFIG_VIRTIO_MMIO) += virtio_mmio.o
>  obj-$(CONFIG_VIRTIO_PCI) += virtio_pci.o
>  virtio_pci-y := virtio_pci_modern.o virtio_pci_common.o
> 
_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V4 2/3] vdpa: set the virtqueue num during register

[Jason Wang]

This patch delay the queue number setting to vDPA device
registering. This allows us to probe the virtqueue numbers between
device allocation and registering.

Reviewed-by: Stefano Garzarella <sgarzare@redhat.com>
Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/vdpa/ifcvf/ifcvf_main.c      |  5 ++---
 drivers/vdpa/mlx5/net/mlx5_vnet.c    |  4 ++--
 drivers/vdpa/vdpa.c                  | 18 ++++++++++--------
 drivers/vdpa/vdpa_sim/vdpa_sim.c     |  2 +-
 drivers/vdpa/vdpa_sim/vdpa_sim_net.c |  2 +-
 include/linux/vdpa.h                 | 10 +++++-----
 6 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/drivers/vdpa/ifcvf/ifcvf_main.c b/drivers/vdpa/ifcvf/ifcvf_main.c
index 7c8bbfcf6c3e..d555a6a5d1ba 100644
--- a/drivers/vdpa/ifcvf/ifcvf_main.c
+++ b/drivers/vdpa/ifcvf/ifcvf_main.c
@@ -431,8 +431,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	}
 
 	adapter = vdpa_alloc_device(struct ifcvf_adapter, vdpa,
-				    dev, &ifc_vdpa_ops,
-				    IFCVF_MAX_QUEUE_PAIRS * 2, NULL);
+				    dev, &ifc_vdpa_ops, NULL);
 	if (adapter == NULL) {
 		IFCVF_ERR(pdev, "Failed to allocate vDPA structure");
 		return -ENOMEM;
@@ -456,7 +455,7 @@ static int ifcvf_probe(struct pci_dev *pdev, const struct pci_device_id *id)
 	for (i = 0; i < IFCVF_MAX_QUEUE_PAIRS * 2; i++)
 		vf->vring[i].irq = -EINVAL;
 
-	ret = vdpa_register_device(&adapter->vdpa);
+	ret = vdpa_register_device(&adapter->vdpa, IFCVF_MAX_QUEUE_PAIRS * 2);
 	if (ret) {
 		IFCVF_ERR(pdev, "Failed to register ifcvf to vdpa bus");
 		goto err;
diff --git a/drivers/vdpa/mlx5/net/mlx5_vnet.c b/drivers/vdpa/mlx5/net/mlx5_vnet.c
index 10e9b09932eb..71397fdafa6a 100644
--- a/drivers/vdpa/mlx5/net/mlx5_vnet.c
+++ b/drivers/vdpa/mlx5/net/mlx5_vnet.c
@@ -1982,7 +1982,7 @@ static int mlx5v_probe(struct auxiliary_device *adev,
 	max_vqs = min_t(u32, max_vqs, MLX5_MAX_SUPPORTED_VQS);
 
 	ndev = vdpa_alloc_device(struct mlx5_vdpa_net, mvdev.vdev, mdev->device, &mlx5_vdpa_ops,
-				 2 * mlx5_vdpa_max_qps(max_vqs), NULL);
+				 NULL);
 	if (IS_ERR(ndev))
 		return PTR_ERR(ndev);
 
@@ -2009,7 +2009,7 @@ static int mlx5v_probe(struct auxiliary_device *adev,
 	if (err)
 		goto err_res;
 
-	err = vdpa_register_device(&mvdev->vdev);
+	err = vdpa_register_device(&mvdev->vdev, 2 * mlx5_vdpa_max_qps(max_vqs));
 	if (err)
 		goto err_reg;
 
diff --git a/drivers/vdpa/vdpa.c b/drivers/vdpa/vdpa.c
index 3d997b389345..55a15c51e243 100644
--- a/drivers/vdpa/vdpa.c
+++ b/drivers/vdpa/vdpa.c
@@ -69,7 +69,6 @@ static void vdpa_release_dev(struct device *d)
  * initialized but before registered.
  * @parent: the parent device
  * @config: the bus operations that is supported by this device
- * @nvqs: number of virtqueues supported by this device
  * @size: size of the parent structure that contains private data
  * @name: name of the vdpa device; optional.
  *
@@ -81,7 +80,7 @@ static void vdpa_release_dev(struct device *d)
  */
 struct vdpa_device *__vdpa_alloc_device(struct device *parent,
 					const struct vdpa_config_ops *config,
-					int nvqs, size_t size, const char *name)
+					size_t size, const char *name)
 {
 	struct vdpa_device *vdev;
 	int err = -EINVAL;
@@ -107,7 +106,6 @@ struct vdpa_device *__vdpa_alloc_device(struct device *parent,
 	vdev->index = err;
 	vdev->config = config;
 	vdev->features_valid = false;
-	vdev->nvqs = nvqs;
 
 	if (name)
 		err = dev_set_name(&vdev->dev, "%s", name);
@@ -136,10 +134,12 @@ static int vdpa_name_match(struct device *dev, const void *data)
 	return (strcmp(dev_name(&vdev->dev), data) == 0);
 }
 
-static int __vdpa_register_device(struct vdpa_device *vdev)
+static int __vdpa_register_device(struct vdpa_device *vdev, int nvqs)
 {
 	struct device *dev;
 
+	vdev->nvqs = nvqs;
+
 	lockdep_assert_held(&vdpa_dev_mutex);
 	dev = bus_find_device(&vdpa_bus, NULL, dev_name(&vdev->dev), vdpa_name_match);
 	if (dev) {
@@ -155,15 +155,16 @@ static int __vdpa_register_device(struct vdpa_device *vdev)
  * Caller must invoke this routine in the management device dev_add()
  * callback after setting up valid mgmtdev for this vdpa device.
  * @vdev: the vdpa device to be registered to vDPA bus
+ * @nvqs: number of virtqueues supported by this device
  *
  * Returns an error when fail to add device to vDPA bus
  */
-int _vdpa_register_device(struct vdpa_device *vdev)
+int _vdpa_register_device(struct vdpa_device *vdev, int nvqs)
 {
 	if (!vdev->mdev)
 		return -EINVAL;
 
-	return __vdpa_register_device(vdev);
+	return __vdpa_register_device(vdev, nvqs);
 }
 EXPORT_SYMBOL_GPL(_vdpa_register_device);
 
@@ -171,15 +172,16 @@ EXPORT_SYMBOL_GPL(_vdpa_register_device);
  * vdpa_register_device - register a vDPA device
  * Callers must have a succeed call of vdpa_alloc_device() before.
  * @vdev: the vdpa device to be registered to vDPA bus
+ * @nvqs: number of virtqueues supported by this device
  *
  * Returns an error when fail to add to vDPA bus
  */
-int vdpa_register_device(struct vdpa_device *vdev)
+int vdpa_register_device(struct vdpa_device *vdev, int nvqs)
 {
 	int err;
 
 	mutex_lock(&vdpa_dev_mutex);
-	err = __vdpa_register_device(vdev);
+	err = __vdpa_register_device(vdev, nvqs);
 	mutex_unlock(&vdpa_dev_mutex);
 	return err;
 }
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim.c b/drivers/vdpa/vdpa_sim/vdpa_sim.c
index d5942842432d..5b6b2f87d40c 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim.c
@@ -235,7 +235,7 @@ struct vdpasim *vdpasim_create(struct vdpasim_dev_attr *dev_attr)
 		ops = &vdpasim_config_ops;
 
 	vdpasim = vdpa_alloc_device(struct vdpasim, vdpa, NULL, ops,
-				    dev_attr->nvqs, dev_attr->name);
+				    dev_attr->name);
 	if (!vdpasim)
 		goto err_alloc;
 
diff --git a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
index d344c5b7c914..702be74877d2 100644
--- a/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
+++ b/drivers/vdpa/vdpa_sim/vdpa_sim_net.c
@@ -147,7 +147,7 @@ static int vdpasim_net_dev_add(struct vdpa_mgmt_dev *mdev, const char *name)
 	if (IS_ERR(simdev))
 		return PTR_ERR(simdev);
 
-	ret = _vdpa_register_device(&simdev->vdpa);
+	ret = _vdpa_register_device(&simdev->vdpa, VDPASIM_NET_VQ_NUM);
 	if (ret)
 		goto reg_err;
 
diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
index 4ab5494503a8..15fa085fab05 100644
--- a/include/linux/vdpa.h
+++ b/include/linux/vdpa.h
@@ -250,20 +250,20 @@ struct vdpa_config_ops {
 
 struct vdpa_device *__vdpa_alloc_device(struct device *parent,
 					const struct vdpa_config_ops *config,
-					int nvqs, size_t size, const char *name);
+					size_t size, const char *name);
 
-#define vdpa_alloc_device(dev_struct, member, parent, config, nvqs, name)   \
+#define vdpa_alloc_device(dev_struct, member, parent, config, name)   \
 			  container_of(__vdpa_alloc_device( \
-				       parent, config, nvqs, \
+				       parent, config, \
 				       sizeof(dev_struct) + \
 				       BUILD_BUG_ON_ZERO(offsetof( \
 				       dev_struct, member)), name), \
 				       dev_struct, member)
 
-int vdpa_register_device(struct vdpa_device *vdev);
+int vdpa_register_device(struct vdpa_device *vdev, int nvqs);
 void vdpa_unregister_device(struct vdpa_device *vdev);
 
-int _vdpa_register_device(struct vdpa_device *vdev);
+int _vdpa_register_device(struct vdpa_device *vdev, int nvqs);
 void _vdpa_unregister_device(struct vdpa_device *vdev);
 
 /**
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

[PATCH V4 3/3] vdpa: introduce virtio pci driver

[Jason Wang]

This patch introduce a vDPA driver for virtio-pci device. It bridges
the virtio-pci control command to the vDPA bus. This will be used for
features prototyping and testing.

Note that get/restore virtqueue state is not supported which needs
extension on the virtio specification.

Signed-off-by: Jason Wang <jasowang@redhat.com>
---
 drivers/vdpa/Kconfig              |   7 +
 drivers/vdpa/Makefile             |   1 +
 drivers/vdpa/virtio_pci/Makefile  |   2 +
 drivers/vdpa/virtio_pci/vp_vdpa.c | 458 ++++++++++++++++++++++++++++++
 4 files changed, 468 insertions(+)
 create mode 100644 drivers/vdpa/virtio_pci/Makefile
 create mode 100644 drivers/vdpa/virtio_pci/vp_vdpa.c

diff --git a/drivers/vdpa/Kconfig b/drivers/vdpa/Kconfig
index ffd1e098bfd2..a245809c99d0 100644
--- a/drivers/vdpa/Kconfig
+++ b/drivers/vdpa/Kconfig
@@ -52,4 +52,11 @@ config MLX5_VDPA_NET
 	  be executed by the hardware. It also supports a variety of stateless
 	  offloads depending on the actual device used and firmware version.
 
+config VP_VDPA
+	tristate "Virtio PCI bridge vDPA driver"
+	select VIRTIO_PCI_LIB
+	depends on PCI_MSI
+	help
+	  This kernel module bridges virtio PCI device to vDPA bus.
+
 endif # VDPA
diff --git a/drivers/vdpa/Makefile b/drivers/vdpa/Makefile
index d160e9b63a66..67fe7f3d6943 100644
--- a/drivers/vdpa/Makefile
+++ b/drivers/vdpa/Makefile
@@ -3,3 +3,4 @@ obj-$(CONFIG_VDPA) += vdpa.o
 obj-$(CONFIG_VDPA_SIM) += vdpa_sim/
 obj-$(CONFIG_IFCVF)    += ifcvf/
 obj-$(CONFIG_MLX5_VDPA) += mlx5/
+obj-$(CONFIG_VP_VDPA)    += virtio_pci/
diff --git a/drivers/vdpa/virtio_pci/Makefile b/drivers/vdpa/virtio_pci/Makefile
new file mode 100644
index 000000000000..231088d3af7d
--- /dev/null
+++ b/drivers/vdpa/virtio_pci/Makefile
@@ -0,0 +1,2 @@
+# SPDX-License-Identifier: GPL-2.0
+obj-$(CONFIG_VP_VDPA) += vp_vdpa.o
diff --git a/drivers/vdpa/virtio_pci/vp_vdpa.c b/drivers/vdpa/virtio_pci/vp_vdpa.c
new file mode 100644
index 000000000000..1321a2fcd088
--- /dev/null
+++ b/drivers/vdpa/virtio_pci/vp_vdpa.c
@@ -0,0 +1,458 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * vDPA bridge driver for modern virtio-pci device
+ *
+ * Copyright (c) 2020, Red Hat Inc. All rights reserved.
+ * Author: Jason Wang <jasowang@redhat.com>
+ *
+ * Based on virtio_pci_modern.c.
+ */
+
+#include <linux/interrupt.h>
+#include <linux/module.h>
+#include <linux/pci.h>
+#include <linux/vdpa.h>
+#include <linux/virtio.h>
+#include <linux/virtio_config.h>
+#include <linux/virtio_ring.h>
+#include <linux/virtio_pci.h>
+#include <linux/virtio_pci_modern.h>
+
+#define VP_VDPA_QUEUE_MAX 256
+#define VP_VDPA_DRIVER_NAME "vp_vdpa"
+#define VP_VDPA_NAME_SIZE 256
+
+struct vp_vring {
+	void __iomem *notify;
+	char msix_name[VP_VDPA_NAME_SIZE];
+	struct vdpa_callback cb;
+	int irq;
+};
+
+struct vp_vdpa {
+	struct vdpa_device vdpa;
+	struct virtio_pci_modern_device mdev;
+	struct vp_vring *vring;
+	struct vdpa_callback config_cb;
+	char msix_name[VP_VDPA_NAME_SIZE];
+	int config_irq;
+	int queues;
+	int vectors;
+};
+
+static struct vp_vdpa *vdpa_to_vp(struct vdpa_device *vdpa)
+{
+	return container_of(vdpa, struct vp_vdpa, vdpa);
+}
+
+static struct virtio_pci_modern_device *vdpa_to_mdev(struct vdpa_device *vdpa)
+{
+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
+
+	return &vp_vdpa->mdev;
+}
+
+static u64 vp_vdpa_get_features(struct vdpa_device *vdpa)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	return vp_modern_get_features(mdev);
+}
+
+static int vp_vdpa_set_features(struct vdpa_device *vdpa, u64 features)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	vp_modern_set_features(mdev, features);
+
+	return 0;
+}
+
+static u8 vp_vdpa_get_status(struct vdpa_device *vdpa)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	return vp_modern_get_status(mdev);
+}
+
+static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
+{
+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
+	struct pci_dev *pdev = mdev->pci_dev;
+	int i;
+
+	for (i = 0; i < vp_vdpa->queues; i++) {
+		if (vp_vdpa->vring[i].irq != VIRTIO_MSI_NO_VECTOR) {
+			vp_modern_queue_vector(mdev, i, VIRTIO_MSI_NO_VECTOR);
+			devm_free_irq(&pdev->dev, vp_vdpa->vring[i].irq,
+				      &vp_vdpa->vring[i]);
+			vp_vdpa->vring[i].irq = VIRTIO_MSI_NO_VECTOR;
+		}
+	}
+
+	if (vp_vdpa->config_irq != VIRTIO_MSI_NO_VECTOR) {
+		vp_modern_config_vector(mdev, VIRTIO_MSI_NO_VECTOR);
+		devm_free_irq(&pdev->dev, vp_vdpa->config_irq, vp_vdpa);
+		vp_vdpa->config_irq = VIRTIO_MSI_NO_VECTOR;
+	}
+
+	if (vp_vdpa->vectors) {
+		pci_free_irq_vectors(pdev);
+		vp_vdpa->vectors = 0;
+	}
+}
+
+static irqreturn_t vp_vdpa_vq_handler(int irq, void *arg)
+{
+	struct vp_vring *vring = arg;
+
+	if (vring->cb.callback)
+		return vring->cb.callback(vring->cb.private);
+
+	return IRQ_HANDLED;
+}
+
+static irqreturn_t vp_vdpa_config_handler(int irq, void *arg)
+{
+	struct vp_vdpa *vp_vdpa = arg;
+
+	if (vp_vdpa->config_cb.callback)
+		return vp_vdpa->config_cb.callback(vp_vdpa->config_cb.private);
+
+	return IRQ_HANDLED;
+}
+
+static int vp_vdpa_request_irq(struct vp_vdpa *vp_vdpa)
+{
+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
+	struct pci_dev *pdev = mdev->pci_dev;
+	int i, ret, irq;
+	int queues = vp_vdpa->queues;
+	int vectors = queues + 1;
+
+	ret = pci_alloc_irq_vectors(pdev, vectors, vectors, PCI_IRQ_MSIX);
+	if (ret != vectors) {
+		dev_err(&pdev->dev,
+			"vp_vdpa: fail to allocate irq vectors want %d but %d\n",
+			vectors, ret);
+		return ret;
+	}
+
+	vp_vdpa->vectors = vectors;
+
+	for (i = 0; i < queues; i++) {
+		snprintf(vp_vdpa->vring[i].msix_name, VP_VDPA_NAME_SIZE,
+			"vp-vdpa[%s]-%d\n", pci_name(pdev), i);
+		irq = pci_irq_vector(pdev, i);
+		ret = devm_request_irq(&pdev->dev, irq,
+				       vp_vdpa_vq_handler,
+				       0, vp_vdpa->vring[i].msix_name,
+				       &vp_vdpa->vring[i]);
+		if (ret) {
+			dev_err(&pdev->dev,
+				"vp_vdpa: fail to request irq for vq %d\n", i);
+			goto err;
+		}
+		vp_modern_queue_vector(mdev, i, i);
+		vp_vdpa->vring[i].irq = irq;
+	}
+
+	snprintf(vp_vdpa->msix_name, VP_VDPA_NAME_SIZE, "vp-vdpa[%s]-config\n",
+		 pci_name(pdev));
+	irq = pci_irq_vector(pdev, queues);
+	ret = devm_request_irq(&pdev->dev, irq,	vp_vdpa_config_handler, 0,
+			       vp_vdpa->msix_name, vp_vdpa);
+	if (ret) {
+		dev_err(&pdev->dev,
+			"vp_vdpa: fail to request irq for vq %d\n", i);
+			goto err;
+	}
+	vp_modern_config_vector(mdev, queues);
+	vp_vdpa->config_irq = irq;
+
+	return 0;
+err:
+	vp_vdpa_free_irq(vp_vdpa);
+	return ret;
+}
+
+static void vp_vdpa_set_status(struct vdpa_device *vdpa, u8 status)
+{
+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
+	u8 s = vp_vdpa_get_status(vdpa);
+
+	if (status & VIRTIO_CONFIG_S_DRIVER_OK &&
+	    !(s & VIRTIO_CONFIG_S_DRIVER_OK)) {
+		vp_vdpa_request_irq(vp_vdpa);
+	}
+
+	vp_modern_set_status(mdev, status);
+
+	if (!(status & VIRTIO_CONFIG_S_DRIVER_OK) &&
+	    (s & VIRTIO_CONFIG_S_DRIVER_OK))
+		vp_vdpa_free_irq(vp_vdpa);
+}
+
+static u16 vp_vdpa_get_vq_num_max(struct vdpa_device *vdpa)
+{
+	return VP_VDPA_QUEUE_MAX;
+}
+
+static int vp_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 qid,
+				struct vdpa_vq_state *state)
+{
+	/* Note that this is not supported by virtio specification, so
+	 * we return -EOPNOTSUPP here. This means we can't support live
+	 * migration, vhost device start/stop.
+	 */
+	return -EOPNOTSUPP;
+}
+
+static int vp_vdpa_set_vq_state(struct vdpa_device *vdpa, u16 qid,
+				const struct vdpa_vq_state *state)
+{
+	/* Note that this is not supported by virtio specification, so
+	 * we return -ENOPOTSUPP here. This means we can't support live
+	 * migration, vhost device start/stop.
+	 */
+	return -EOPNOTSUPP;
+}
+
+static void vp_vdpa_set_vq_cb(struct vdpa_device *vdpa, u16 qid,
+			      struct vdpa_callback *cb)
+{
+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
+
+	vp_vdpa->vring[qid].cb = *cb;
+}
+
+static void vp_vdpa_set_vq_ready(struct vdpa_device *vdpa,
+				 u16 qid, bool ready)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	vp_modern_set_queue_enable(mdev, qid, ready);
+}
+
+static bool vp_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 qid)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	return vp_modern_get_queue_enable(mdev, qid);
+}
+
+static void vp_vdpa_set_vq_num(struct vdpa_device *vdpa, u16 qid,
+			       u32 num)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	vp_modern_set_queue_size(mdev, qid, num);
+}
+
+static int vp_vdpa_set_vq_address(struct vdpa_device *vdpa, u16 qid,
+				  u64 desc_area, u64 driver_area,
+				  u64 device_area)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	vp_modern_queue_address(mdev, qid, desc_area,
+				driver_area, device_area);
+
+	return 0;
+}
+
+static void vp_vdpa_kick_vq(struct vdpa_device *vdpa, u16 qid)
+{
+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
+
+	vp_iowrite16(qid, vp_vdpa->vring[qid].notify);
+}
+
+static u32 vp_vdpa_get_generation(struct vdpa_device *vdpa)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	return vp_modern_generation(mdev);
+}
+
+static u32 vp_vdpa_get_device_id(struct vdpa_device *vdpa)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	return mdev->id.device;
+}
+
+static u32 vp_vdpa_get_vendor_id(struct vdpa_device *vdpa)
+{
+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
+
+	return mdev->id.vendor;
+}
+
+static u32 vp_vdpa_get_vq_align(struct vdpa_device *vdpa)
+{
+	return PAGE_SIZE;
+}
+
+static void vp_vdpa_get_config(struct vdpa_device *vdpa,
+			       unsigned int offset,
+			       void *buf, unsigned int len)
+{
+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
+	u8 old, new;
+	u8 *p;
+	int i;
+
+	do {
+		old = vp_ioread8(&mdev->common->config_generation);
+		p = buf;
+		for (i = 0; i < len; i++)
+			*p++ = vp_ioread8(mdev->device + offset + i);
+
+		new = vp_ioread8(&mdev->common->config_generation);
+	} while (old != new);
+}
+
+static void vp_vdpa_set_config(struct vdpa_device *vdpa,
+			       unsigned int offset, const void *buf,
+			       unsigned int len)
+{
+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
+	const u8 *p = buf;
+	int i;
+
+	for (i = 0; i < len; i++)
+		vp_iowrite8(*p++, mdev->device + offset + i);
+}
+
+static void vp_vdpa_set_config_cb(struct vdpa_device *vdpa,
+				  struct vdpa_callback *cb)
+{
+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
+
+	vp_vdpa->config_cb = *cb;
+}
+
+static const struct vdpa_config_ops vp_vdpa_ops = {
+	.get_features	= vp_vdpa_get_features,
+	.set_features	= vp_vdpa_set_features,
+	.get_status	= vp_vdpa_get_status,
+	.set_status	= vp_vdpa_set_status,
+	.get_vq_num_max	= vp_vdpa_get_vq_num_max,
+	.get_vq_state	= vp_vdpa_get_vq_state,
+	.set_vq_state	= vp_vdpa_set_vq_state,
+	.set_vq_cb	= vp_vdpa_set_vq_cb,
+	.set_vq_ready	= vp_vdpa_set_vq_ready,
+	.get_vq_ready	= vp_vdpa_get_vq_ready,
+	.set_vq_num	= vp_vdpa_set_vq_num,
+	.set_vq_address	= vp_vdpa_set_vq_address,
+	.kick_vq	= vp_vdpa_kick_vq,
+	.get_generation	= vp_vdpa_get_generation,
+	.get_device_id	= vp_vdpa_get_device_id,
+	.get_vendor_id	= vp_vdpa_get_vendor_id,
+	.get_vq_align	= vp_vdpa_get_vq_align,
+	.get_config	= vp_vdpa_get_config,
+	.set_config	= vp_vdpa_set_config,
+	.set_config_cb  = vp_vdpa_set_config_cb,
+};
+
+static void vp_vdpa_free_irq_vectors(void *data)
+{
+	pci_free_irq_vectors(data);
+}
+
+static int vp_vdpa_probe(struct pci_dev *pdev, const struct pci_device_id *id)
+{
+	struct virtio_pci_modern_device *mdev;
+	struct device *dev = &pdev->dev;
+	struct vp_vdpa *vp_vdpa;
+	u16 notify_off;
+	int ret, i;
+
+	ret = pcim_enable_device(pdev);
+	if (ret)
+		return ret;
+
+	vp_vdpa = vdpa_alloc_device(struct vp_vdpa, vdpa,
+				    dev, &vp_vdpa_ops, NULL);
+	if (vp_vdpa == NULL) {
+		dev_err(dev, "vp_vdpa: Failed to allocate vDPA structure\n");
+		return -ENOMEM;
+	}
+
+	mdev = &vp_vdpa->mdev;
+	mdev->pci_dev = pdev;
+
+	ret = vp_modern_probe(mdev);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed to probe modern PCI device\n");
+		goto err;
+	}
+
+	pci_set_master(pdev);
+	pci_set_drvdata(pdev, vp_vdpa);
+
+	vp_vdpa->vdpa.dma_dev = &pdev->dev;
+	vp_vdpa->queues = vp_modern_get_num_queues(mdev);
+
+	ret = devm_add_action_or_reset(dev, vp_vdpa_free_irq_vectors, pdev);
+	if (ret) {
+		dev_err(&pdev->dev,
+			"Failed for adding devres for freeing irq vectors\n");
+		goto err;
+	}
+
+	vp_vdpa->vring = devm_kcalloc(&pdev->dev, vp_vdpa->queues,
+				      sizeof(*vp_vdpa->vring),
+				      GFP_KERNEL);
+	if (!vp_vdpa->vring) {
+		ret = -ENOMEM;
+		dev_err(&pdev->dev, "Fail to allocate virtqueues\n");
+		goto err;
+	}
+
+	for (i = 0; i < vp_vdpa->queues; i++) {
+		notify_off = vp_modern_get_queue_notify_off(mdev, i);
+		vp_vdpa->vring[i].irq = VIRTIO_MSI_NO_VECTOR;
+		vp_vdpa->vring[i].notify = mdev->notify_base +
+			notify_off * mdev->notify_offset_multiplier;
+	}
+	vp_vdpa->config_irq = VIRTIO_MSI_NO_VECTOR;
+
+	ret = vdpa_register_device(&vp_vdpa->vdpa, vp_vdpa->queues);
+	if (ret) {
+		dev_err(&pdev->dev, "Failed to register to vdpa bus\n");
+		goto err;
+	}
+
+	return 0;
+
+err:
+	put_device(&vp_vdpa->vdpa.dev);
+	return ret;
+}
+
+static void vp_vdpa_remove(struct pci_dev *pdev)
+{
+	struct vp_vdpa *vp_vdpa = pci_get_drvdata(pdev);
+
+	vdpa_unregister_device(&vp_vdpa->vdpa);
+	vp_modern_remove(&vp_vdpa->mdev);
+}
+
+static struct pci_driver vp_vdpa_driver = {
+	.name		= "vp-vdpa",
+	.id_table	= NULL, /* only dynamic ids */
+	.probe		= vp_vdpa_probe,
+	.remove		= vp_vdpa_remove,
+};
+
+module_pci_driver(vp_vdpa_driver);
+
+MODULE_AUTHOR("Jason Wang <jasowang@redhat.com>");
+MODULE_DESCRIPTION("vp-vdpa");
+MODULE_LICENSE("GPL");
+MODULE_VERSION("1");
-- 
2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 3/3] vdpa: introduce virtio pci driver

[Stefano Garzarella]

On Tue, Feb 23, 2021 at 02:19:05PM +0800, Jason Wang wrote:
>This patch introduce a vDPA driver for virtio-pci device. It bridges
>the virtio-pci control command to the vDPA bus. This will be used for
>features prototyping and testing.
>
>Note that get/restore virtqueue state is not supported which needs
>extension on the virtio specification.
>
>Signed-off-by: Jason Wang <jasowang@redhat.com>
>---
> drivers/vdpa/Kconfig              |   7 +
> drivers/vdpa/Makefile             |   1 +
> drivers/vdpa/virtio_pci/Makefile  |   2 +
> drivers/vdpa/virtio_pci/vp_vdpa.c | 458 ++++++++++++++++++++++++++++++
> 4 files changed, 468 insertions(+)
> create mode 100644 drivers/vdpa/virtio_pci/Makefile
> create mode 100644 drivers/vdpa/virtio_pci/vp_vdpa.c

Acked-by: Stefano Garzarella <sgarzare@redhat.com>

>
>diff --git a/drivers/vdpa/Kconfig b/drivers/vdpa/Kconfig
>index ffd1e098bfd2..a245809c99d0 100644
>--- a/drivers/vdpa/Kconfig
>+++ b/drivers/vdpa/Kconfig
>@@ -52,4 +52,11 @@ config MLX5_VDPA_NET
> 	  be executed by the hardware. It also supports a variety of stateless
> 	  offloads depending on the actual device used and firmware version.
>
>+config VP_VDPA
>+	tristate "Virtio PCI bridge vDPA driver"
>+	select VIRTIO_PCI_LIB
>+	depends on PCI_MSI
>+	help
>+	  This kernel module bridges virtio PCI device to vDPA bus.
>+
> endif # VDPA
>diff --git a/drivers/vdpa/Makefile b/drivers/vdpa/Makefile
>index d160e9b63a66..67fe7f3d6943 100644
>--- a/drivers/vdpa/Makefile
>+++ b/drivers/vdpa/Makefile
>@@ -3,3 +3,4 @@ obj-$(CONFIG_VDPA) += vdpa.o
> obj-$(CONFIG_VDPA_SIM) += vdpa_sim/
> obj-$(CONFIG_IFCVF)    += ifcvf/
> obj-$(CONFIG_MLX5_VDPA) += mlx5/
>+obj-$(CONFIG_VP_VDPA)    += virtio_pci/
>diff --git a/drivers/vdpa/virtio_pci/Makefile b/drivers/vdpa/virtio_pci/Makefile
>new file mode 100644
>index 000000000000..231088d3af7d
>--- /dev/null
>+++ b/drivers/vdpa/virtio_pci/Makefile
>@@ -0,0 +1,2 @@
>+# SPDX-License-Identifier: GPL-2.0
>+obj-$(CONFIG_VP_VDPA) += vp_vdpa.o
>diff --git a/drivers/vdpa/virtio_pci/vp_vdpa.c b/drivers/vdpa/virtio_pci/vp_vdpa.c
>new file mode 100644
>index 000000000000..1321a2fcd088
>--- /dev/null
>+++ b/drivers/vdpa/virtio_pci/vp_vdpa.c
>@@ -0,0 +1,458 @@
>+// SPDX-License-Identifier: GPL-2.0-only
>+/*
>+ * vDPA bridge driver for modern virtio-pci device
>+ *
>+ * Copyright (c) 2020, Red Hat Inc. All rights reserved.
>+ * Author: Jason Wang <jasowang@redhat.com>
>+ *
>+ * Based on virtio_pci_modern.c.
>+ */
>+
>+#include <linux/interrupt.h>
>+#include <linux/module.h>
>+#include <linux/pci.h>
>+#include <linux/vdpa.h>
>+#include <linux/virtio.h>
>+#include <linux/virtio_config.h>
>+#include <linux/virtio_ring.h>
>+#include <linux/virtio_pci.h>
>+#include <linux/virtio_pci_modern.h>
>+
>+#define VP_VDPA_QUEUE_MAX 256
>+#define VP_VDPA_DRIVER_NAME "vp_vdpa"
>+#define VP_VDPA_NAME_SIZE 256
>+
>+struct vp_vring {
>+	void __iomem *notify;
>+	char msix_name[VP_VDPA_NAME_SIZE];
>+	struct vdpa_callback cb;
>+	int irq;
>+};
>+
>+struct vp_vdpa {
>+	struct vdpa_device vdpa;
>+	struct virtio_pci_modern_device mdev;
>+	struct vp_vring *vring;
>+	struct vdpa_callback config_cb;
>+	char msix_name[VP_VDPA_NAME_SIZE];
>+	int config_irq;
>+	int queues;
>+	int vectors;
>+};
>+
>+static struct vp_vdpa *vdpa_to_vp(struct vdpa_device *vdpa)
>+{
>+	return container_of(vdpa, struct vp_vdpa, vdpa);
>+}
>+
>+static struct virtio_pci_modern_device *vdpa_to_mdev(struct vdpa_device *vdpa)
>+{
>+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
>+
>+	return &vp_vdpa->mdev;
>+}
>+
>+static u64 vp_vdpa_get_features(struct vdpa_device *vdpa)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	return vp_modern_get_features(mdev);
>+}
>+
>+static int vp_vdpa_set_features(struct vdpa_device *vdpa, u64 features)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	vp_modern_set_features(mdev, features);
>+
>+	return 0;
>+}
>+
>+static u8 vp_vdpa_get_status(struct vdpa_device *vdpa)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	return vp_modern_get_status(mdev);
>+}
>+
>+static void vp_vdpa_free_irq(struct vp_vdpa *vp_vdpa)
>+{
>+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
>+	struct pci_dev *pdev = mdev->pci_dev;
>+	int i;
>+
>+	for (i = 0; i < vp_vdpa->queues; i++) {
>+		if (vp_vdpa->vring[i].irq != VIRTIO_MSI_NO_VECTOR) {
>+			vp_modern_queue_vector(mdev, i, VIRTIO_MSI_NO_VECTOR);
>+			devm_free_irq(&pdev->dev, vp_vdpa->vring[i].irq,
>+				      &vp_vdpa->vring[i]);
>+			vp_vdpa->vring[i].irq = VIRTIO_MSI_NO_VECTOR;
>+		}
>+	}
>+
>+	if (vp_vdpa->config_irq != VIRTIO_MSI_NO_VECTOR) {
>+		vp_modern_config_vector(mdev, VIRTIO_MSI_NO_VECTOR);
>+		devm_free_irq(&pdev->dev, vp_vdpa->config_irq, vp_vdpa);
>+		vp_vdpa->config_irq = VIRTIO_MSI_NO_VECTOR;
>+	}
>+
>+	if (vp_vdpa->vectors) {
>+		pci_free_irq_vectors(pdev);
>+		vp_vdpa->vectors = 0;
>+	}
>+}
>+
>+static irqreturn_t vp_vdpa_vq_handler(int irq, void *arg)
>+{
>+	struct vp_vring *vring = arg;
>+
>+	if (vring->cb.callback)
>+		return vring->cb.callback(vring->cb.private);
>+
>+	return IRQ_HANDLED;
>+}
>+
>+static irqreturn_t vp_vdpa_config_handler(int irq, void *arg)
>+{
>+	struct vp_vdpa *vp_vdpa = arg;
>+
>+	if (vp_vdpa->config_cb.callback)
>+		return vp_vdpa->config_cb.callback(vp_vdpa->config_cb.private);
>+
>+	return IRQ_HANDLED;
>+}
>+
>+static int vp_vdpa_request_irq(struct vp_vdpa *vp_vdpa)
>+{
>+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
>+	struct pci_dev *pdev = mdev->pci_dev;
>+	int i, ret, irq;
>+	int queues = vp_vdpa->queues;
>+	int vectors = queues + 1;
>+
>+	ret = pci_alloc_irq_vectors(pdev, vectors, vectors, PCI_IRQ_MSIX);
>+	if (ret != vectors) {
>+		dev_err(&pdev->dev,
>+			"vp_vdpa: fail to allocate irq vectors want %d but %d\n",
>+			vectors, ret);
>+		return ret;
>+	}
>+
>+	vp_vdpa->vectors = vectors;
>+
>+	for (i = 0; i < queues; i++) {
>+		snprintf(vp_vdpa->vring[i].msix_name, VP_VDPA_NAME_SIZE,
>+			"vp-vdpa[%s]-%d\n", pci_name(pdev), i);
>+		irq = pci_irq_vector(pdev, i);
>+		ret = devm_request_irq(&pdev->dev, irq,
>+				       vp_vdpa_vq_handler,
>+				       0, vp_vdpa->vring[i].msix_name,
>+				       &vp_vdpa->vring[i]);
>+		if (ret) {
>+			dev_err(&pdev->dev,
>+				"vp_vdpa: fail to request irq for vq %d\n", i);
>+			goto err;
>+		}
>+		vp_modern_queue_vector(mdev, i, i);
>+		vp_vdpa->vring[i].irq = irq;
>+	}
>+
>+	snprintf(vp_vdpa->msix_name, VP_VDPA_NAME_SIZE, "vp-vdpa[%s]-config\n",
>+		 pci_name(pdev));
>+	irq = pci_irq_vector(pdev, queues);
>+	ret = devm_request_irq(&pdev->dev, irq,	vp_vdpa_config_handler, 0,
>+			       vp_vdpa->msix_name, vp_vdpa);
>+	if (ret) {
>+		dev_err(&pdev->dev,
>+			"vp_vdpa: fail to request irq for vq %d\n", i);
>+			goto err;
>+	}
>+	vp_modern_config_vector(mdev, queues);
>+	vp_vdpa->config_irq = irq;
>+
>+	return 0;
>+err:
>+	vp_vdpa_free_irq(vp_vdpa);
>+	return ret;
>+}
>+
>+static void vp_vdpa_set_status(struct vdpa_device *vdpa, u8 status)
>+{
>+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
>+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
>+	u8 s = vp_vdpa_get_status(vdpa);
>+
>+	if (status & VIRTIO_CONFIG_S_DRIVER_OK &&
>+	    !(s & VIRTIO_CONFIG_S_DRIVER_OK)) {
>+		vp_vdpa_request_irq(vp_vdpa);
>+	}
>+
>+	vp_modern_set_status(mdev, status);
>+
>+	if (!(status & VIRTIO_CONFIG_S_DRIVER_OK) &&
>+	    (s & VIRTIO_CONFIG_S_DRIVER_OK))
>+		vp_vdpa_free_irq(vp_vdpa);
>+}
>+
>+static u16 vp_vdpa_get_vq_num_max(struct vdpa_device *vdpa)
>+{
>+	return VP_VDPA_QUEUE_MAX;
>+}
>+
>+static int vp_vdpa_get_vq_state(struct vdpa_device *vdpa, u16 qid,
>+				struct vdpa_vq_state *state)
>+{
>+	/* Note that this is not supported by virtio specification, so
>+	 * we return -EOPNOTSUPP here. This means we can't support live
>+	 * migration, vhost device start/stop.
>+	 */
>+	return -EOPNOTSUPP;
>+}
>+
>+static int vp_vdpa_set_vq_state(struct vdpa_device *vdpa, u16 qid,
>+				const struct vdpa_vq_state *state)
>+{
>+	/* Note that this is not supported by virtio specification, so
>+	 * we return -ENOPOTSUPP here. This means we can't support live
>+	 * migration, vhost device start/stop.
>+	 */
>+	return -EOPNOTSUPP;
>+}
>+
>+static void vp_vdpa_set_vq_cb(struct vdpa_device *vdpa, u16 qid,
>+			      struct vdpa_callback *cb)
>+{
>+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
>+
>+	vp_vdpa->vring[qid].cb = *cb;
>+}
>+
>+static void vp_vdpa_set_vq_ready(struct vdpa_device *vdpa,
>+				 u16 qid, bool ready)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	vp_modern_set_queue_enable(mdev, qid, ready);
>+}
>+
>+static bool vp_vdpa_get_vq_ready(struct vdpa_device *vdpa, u16 qid)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	return vp_modern_get_queue_enable(mdev, qid);
>+}
>+
>+static void vp_vdpa_set_vq_num(struct vdpa_device *vdpa, u16 qid,
>+			       u32 num)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	vp_modern_set_queue_size(mdev, qid, num);
>+}
>+
>+static int vp_vdpa_set_vq_address(struct vdpa_device *vdpa, u16 qid,
>+				  u64 desc_area, u64 driver_area,
>+				  u64 device_area)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	vp_modern_queue_address(mdev, qid, desc_area,
>+				driver_area, device_area);
>+
>+	return 0;
>+}
>+
>+static void vp_vdpa_kick_vq(struct vdpa_device *vdpa, u16 qid)
>+{
>+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
>+
>+	vp_iowrite16(qid, vp_vdpa->vring[qid].notify);
>+}
>+
>+static u32 vp_vdpa_get_generation(struct vdpa_device *vdpa)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	return vp_modern_generation(mdev);
>+}
>+
>+static u32 vp_vdpa_get_device_id(struct vdpa_device *vdpa)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	return mdev->id.device;
>+}
>+
>+static u32 vp_vdpa_get_vendor_id(struct vdpa_device *vdpa)
>+{
>+	struct virtio_pci_modern_device *mdev = vdpa_to_mdev(vdpa);
>+
>+	return mdev->id.vendor;
>+}
>+
>+static u32 vp_vdpa_get_vq_align(struct vdpa_device *vdpa)
>+{
>+	return PAGE_SIZE;
>+}
>+
>+static void vp_vdpa_get_config(struct vdpa_device *vdpa,
>+			       unsigned int offset,
>+			       void *buf, unsigned int len)
>+{
>+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
>+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
>+	u8 old, new;
>+	u8 *p;
>+	int i;
>+
>+	do {
>+		old = vp_ioread8(&mdev->common->config_generation);
>+		p = buf;
>+		for (i = 0; i < len; i++)
>+			*p++ = vp_ioread8(mdev->device + offset + i);
>+
>+		new = vp_ioread8(&mdev->common->config_generation);
>+	} while (old != new);
>+}
>+
>+static void vp_vdpa_set_config(struct vdpa_device *vdpa,
>+			       unsigned int offset, const void *buf,
>+			       unsigned int len)
>+{
>+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
>+	struct virtio_pci_modern_device *mdev = &vp_vdpa->mdev;
>+	const u8 *p = buf;
>+	int i;
>+
>+	for (i = 0; i < len; i++)
>+		vp_iowrite8(*p++, mdev->device + offset + i);
>+}
>+
>+static void vp_vdpa_set_config_cb(struct vdpa_device *vdpa,
>+				  struct vdpa_callback *cb)
>+{
>+	struct vp_vdpa *vp_vdpa = vdpa_to_vp(vdpa);
>+
>+	vp_vdpa->config_cb = *cb;
>+}
>+
>+static const struct vdpa_config_ops vp_vdpa_ops = {
>+	.get_features	= vp_vdpa_get_features,
>+	.set_features	= vp_vdpa_set_features,
>+	.get_status	= vp_vdpa_get_status,
>+	.set_status	= vp_vdpa_set_status,
>+	.get_vq_num_max	= vp_vdpa_get_vq_num_max,
>+	.get_vq_state	= vp_vdpa_get_vq_state,
>+	.set_vq_state	= vp_vdpa_set_vq_state,
>+	.set_vq_cb	= vp_vdpa_set_vq_cb,
>+	.set_vq_ready	= vp_vdpa_set_vq_ready,
>+	.get_vq_ready	= vp_vdpa_get_vq_ready,
>+	.set_vq_num	= vp_vdpa_set_vq_num,
>+	.set_vq_address	= vp_vdpa_set_vq_address,
>+	.kick_vq	= vp_vdpa_kick_vq,
>+	.get_generation	= vp_vdpa_get_generation,
>+	.get_device_id	= vp_vdpa_get_device_id,
>+	.get_vendor_id	= vp_vdpa_get_vendor_id,
>+	.get_vq_align	= vp_vdpa_get_vq_align,
>+	.get_config	= vp_vdpa_get_config,
>+	.set_config	= vp_vdpa_set_config,
>+	.set_config_cb  = vp_vdpa_set_config_cb,
>+};
>+
>+static void vp_vdpa_free_irq_vectors(void *data)
>+{
>+	pci_free_irq_vectors(data);
>+}
>+
>+static int vp_vdpa_probe(struct pci_dev *pdev, const struct pci_device_id *id)
>+{
>+	struct virtio_pci_modern_device *mdev;
>+	struct device *dev = &pdev->dev;
>+	struct vp_vdpa *vp_vdpa;
>+	u16 notify_off;
>+	int ret, i;
>+
>+	ret = pcim_enable_device(pdev);
>+	if (ret)
>+		return ret;
>+
>+	vp_vdpa = vdpa_alloc_device(struct vp_vdpa, vdpa,
>+				    dev, &vp_vdpa_ops, NULL);
>+	if (vp_vdpa == NULL) {
>+		dev_err(dev, "vp_vdpa: Failed to allocate vDPA structure\n");
>+		return -ENOMEM;
>+	}
>+
>+	mdev = &vp_vdpa->mdev;
>+	mdev->pci_dev = pdev;
>+
>+	ret = vp_modern_probe(mdev);
>+	if (ret) {
>+		dev_err(&pdev->dev, "Failed to probe modern PCI device\n");
>+		goto err;
>+	}
>+
>+	pci_set_master(pdev);
>+	pci_set_drvdata(pdev, vp_vdpa);
>+
>+	vp_vdpa->vdpa.dma_dev = &pdev->dev;
>+	vp_vdpa->queues = vp_modern_get_num_queues(mdev);
>+
>+	ret = devm_add_action_or_reset(dev, vp_vdpa_free_irq_vectors, pdev);
>+	if (ret) {
>+		dev_err(&pdev->dev,
>+			"Failed for adding devres for freeing irq vectors\n");
>+		goto err;
>+	}
>+
>+	vp_vdpa->vring = devm_kcalloc(&pdev->dev, vp_vdpa->queues,
>+				      sizeof(*vp_vdpa->vring),
>+				      GFP_KERNEL);
>+	if (!vp_vdpa->vring) {
>+		ret = -ENOMEM;
>+		dev_err(&pdev->dev, "Fail to allocate virtqueues\n");
>+		goto err;
>+	}
>+
>+	for (i = 0; i < vp_vdpa->queues; i++) {
>+		notify_off = vp_modern_get_queue_notify_off(mdev, i);
>+		vp_vdpa->vring[i].irq = VIRTIO_MSI_NO_VECTOR;
>+		vp_vdpa->vring[i].notify = mdev->notify_base +
>+			notify_off * mdev->notify_offset_multiplier;
>+	}
>+	vp_vdpa->config_irq = VIRTIO_MSI_NO_VECTOR;
>+
>+	ret = vdpa_register_device(&vp_vdpa->vdpa, vp_vdpa->queues);
>+	if (ret) {
>+		dev_err(&pdev->dev, "Failed to register to vdpa bus\n");
>+		goto err;
>+	}
>+
>+	return 0;
>+
>+err:
>+	put_device(&vp_vdpa->vdpa.dev);
>+	return ret;
>+}
>+
>+static void vp_vdpa_remove(struct pci_dev *pdev)
>+{
>+	struct vp_vdpa *vp_vdpa = pci_get_drvdata(pdev);
>+
>+	vdpa_unregister_device(&vp_vdpa->vdpa);
>+	vp_modern_remove(&vp_vdpa->mdev);
>+}
>+
>+static struct pci_driver vp_vdpa_driver = {
>+	.name		= "vp-vdpa",
>+	.id_table	= NULL, /* only dynamic ids */
>+	.probe		= vp_vdpa_probe,
>+	.remove		= vp_vdpa_remove,
>+};
>+
>+module_pci_driver(vp_vdpa_driver);
>+
>+MODULE_AUTHOR("Jason Wang <jasowang@redhat.com>");
>+MODULE_DESCRIPTION("vp-vdpa");
>+MODULE_LICENSE("GPL");
>+MODULE_VERSION("1");
>-- 
>2.25.1
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [PATCH V4 0/3] virtio-pci: introduce modern device module

[Michael S. Tsirkin]

On Tue, Feb 23, 2021 at 02:19:02PM +0800, Jason Wang wrote:
> Hi all:
> 
> This series tries to implement a vDPA driver for virtio-pci device
> which will bridge between vDPA bus and virtio-pci device.
> 
> This could be used for future feature prototyping and testing.
> 
> Please review

OK just one thing: subject should have had V5, not V4.


> Changes since V4:
> - include the patch to let VIRTIO_PCI_LIB to be auto selected
> - style fixes in the Kconfig for vp-vdpa driver
> - fix the err value returned during vp-vdpa driver probing
> 
> Changes since V3:
> - rebase to vhost.git
> 
> Changes since V2:
> 
> - tweak config prompt
> - switch from 'cb' to 'config_cb' for vp_vdpa config interrupt
> - use a macro for vp_vdpa msix name length
> 
> Changes since V1:
> 
> - don't try to use devres for virtio-pci core
> - tweak the commit log
> - split the patches furtherly to ease the reviewing
> 
> Changes since RFC:
> 
> - Split common codes from virito-pci and share it with vDPA driver
> - Use dynamic id in order to be less confusing with virtio-pci driver
> - No feature whitelist, supporting any features (mq, config etc)
> 
> Jason Wang (3):
>   virtio: don't prompt CONFIG_VIRTIO_PCI_MODERN
>   vdpa: set the virtqueue num during register
>   vdpa: introduce virtio pci driver
> 
>  drivers/vdpa/Kconfig                 |   7 +
>  drivers/vdpa/Makefile                |   1 +
>  drivers/vdpa/ifcvf/ifcvf_main.c      |   5 +-
>  drivers/vdpa/mlx5/net/mlx5_vnet.c    |   4 +-
>  drivers/vdpa/vdpa.c                  |  18 +-
>  drivers/vdpa/vdpa_sim/vdpa_sim.c     |   2 +-
>  drivers/vdpa/vdpa_sim/vdpa_sim_net.c |   2 +-
>  drivers/vdpa/virtio_pci/Makefile     |   2 +
>  drivers/vdpa/virtio_pci/vp_vdpa.c    | 458 +++++++++++++++++++++++++++
>  drivers/virtio/Kconfig               |  11 +-
>  drivers/virtio/Makefile              |   2 +-
>  include/linux/vdpa.h                 |  10 +-
>  12 files changed, 496 insertions(+), 26 deletions(-)
>  create mode 100644 drivers/vdpa/virtio_pci/Makefile
>  create mode 100644 drivers/vdpa/virtio_pci/vp_vdpa.c
> 
> -- 
> 2.25.1

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization