Re: [RFC 0/7] vdpa: Add support for iommufd

Re: [RFC 0/7] vdpa: Add support for iommufd

[Jason Wang]

On Sun, Sep 24, 2023 at 1:05 AM Cindy Lu <lulu@redhat.com> wrote:
>
> Hi All
> Really apologize for the delay, this is the draft RFC for
> iommufd support for vdpa, This code provides the basic function
> for iommufd support
>
> The code was tested and passed in device vdpa_sim_net
> The qemu code is
> https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> The kernel code is
> https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
>
> ToDo
> 1. this code is out of date and needs to clean and rebase on the latest code
> 2. this code has some workaround, I Skip the check for
> iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> mutex for iommfd operations
> 3. only test in emulated device, other modes not tested yet
>
> After addressed these problems I will send out a new version for RFC. I will
> provide the code in 3 weeks

Something more needs to be done after a quick glance at the codes.

1) The support for device with platform IOMMU support
2) The support for multiple ASes per device

...

Thanks

>
> Thanks
> Cindy
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> The test step is
> 1. create vdpa_sim device
> ...
> vdpa dev add name vdpa15 mgmtdev vdpasim_net
> ...
> 2. load the VM with the command
>   -object iommufd,id=iommufd0 \
>   -device virtio-net-pci,netdev=vhost-vdpa1,disable-legacy=on,disable-modern=off\
>   -netdev type=vhost-vdpa,vhostdev=/dev/vhost-vdpa-0,id=vhost-vdpa1,iommufd=iommufd0\
>
> 3. in guest VM you can find the vdpa_sim port works well.
> [root@ubuntunew ~]# ifconfig eth0
> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet6 fe80::5054:ff:fe12:3456  prefixlen 64  scopeid 0x20<link>
>         ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
>         RX packets 53  bytes 9108 (8.8 KiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 53  bytes 9108 (8.8 KiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>
> [root@ubuntunew ~]# ./test.sh eth0
> [  172.815279] pktgen: Packet Generator for packet performance testing. Version: 2.75
> Adding queue 0 of eth0
> Configuring devices eth0@0
> Running... ctrl^C to stop
>
> [root@ubuntunew ~]# ifconfig eth0
> eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
>         inet6 fe80::5054:ff:fe12:3456  prefixlen 64  scopeid 0x20<link>
>         ether 52:54:00:12:34:56  txqueuelen 1000  (Ethernet)
>         RX packets 183455  bytes 11748533 (11.2 MiB)
>         RX errors 0  dropped 0  overruns 0  frame 0
>         TX packets 183473  bytes 11749685 (11.2 MiB)
>         TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
>
> Cindy Lu (7):
>   vhost/iommufd: Add the functions support iommufd
>   Kconfig: Add the new file vhost/iommufd
>   vhost: Add 3 new uapi to support iommufd
>   vdpa: change the map/unmap process to support iommufd
>   vdpa: Add new vdpa_config_ops
>   vdpa_sim :Add support for iommufd
>   iommufd: Skip the CACHE_COHERENCY and iommu group check
>
>  drivers/iommu/iommufd/device.c   |   6 +-
>  drivers/vdpa/vdpa_sim/vdpa_sim.c |   8 ++
>  drivers/vhost/Kconfig            |   1 +
>  drivers/vhost/Makefile           |   1 +
>  drivers/vhost/iommufd.c          | 151 +++++++++++++++++++++++
>  drivers/vhost/vdpa.c             | 201 +++++++++++++++++++++++++++++++
>  drivers/vhost/vhost.h            |  21 ++++
>  include/linux/vdpa.h             |  34 +++++-
>  include/uapi/linux/vhost.h       |  71 +++++++++++
>  9 files changed, 490 insertions(+), 4 deletions(-)
>  create mode 100644 drivers/vhost/iommufd.c
>
> --
> 2.34.3
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC 5/7] vdpa: Add new vdpa_config_ops

[Jason Wang]

On Sun, Sep 24, 2023 at 1:06 AM Cindy Lu <lulu@redhat.com> wrote:
>
> Add new vdpa_config_ops to support iommufd
>
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> ---
>  include/linux/vdpa.h | 34 +++++++++++++++++++++++++++++++++-
>  1 file changed, 33 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/vdpa.h b/include/linux/vdpa.h
> index 6d0f5e4e82c2..4ada5bd6f90e 100644
> --- a/include/linux/vdpa.h
> +++ b/include/linux/vdpa.h
> @@ -5,6 +5,7 @@
>  #include <linux/kernel.h>
>  #include <linux/device.h>
>  #include <linux/interrupt.h>
> +#include <linux/iommufd.h>
>  #include <linux/vhost_iotlb.h>
>  #include <linux/virtio_net.h>
>  #include <linux/if_ether.h>
> @@ -91,6 +92,12 @@ struct vdpa_device {
>         struct vdpa_mgmt_dev *mdev;
>         unsigned int ngroups;
>         unsigned int nas;
> +       struct iommufd_access *iommufd_access;
> +       struct iommufd_device *iommufd_device;
> +       struct iommufd_ctx *iommufd_ictx;
> +       unsigned long *vq_bitmap;
> +       atomic_t iommufd_users;
> +       bool iommufd_attached;
>  };
>
>  /**
> @@ -282,6 +289,15 @@ struct vdpa_map_file {
>   *                             @iova: iova to be unmapped
>   *                             @size: size of the area
>   *                             Returns integer: success (0) or error (< 0)
> + * @bind_iommufd:              use vdpa_iommufd_physical_bind for an IOMMU
> + *                             backed device.
> + *                             otherwise use vdpa_iommufd_emulated_bind
> + * @unbind_iommufd:            use vdpa_iommufd_physical_unbind for an IOMMU
> + *                             backed device.
> + *                             otherwise, use vdpa_iommufd_emulated_unbind
> + * @attach_ioas:               use vdpa_iommufd_physical_attach_ioas for an
> + *                             IOMMU backed device.
> + * @detach_ioas:               Opposite of attach_ioas

Those should be marked as mandatory only for parents with specific
translations (e.g simulator and mlx5_vdpa).

Or anything I missed?

Thanks


>   * @free:                      Free resources that belongs to vDPA (optional)
>   *                             @vdev: vdpa device
>   */
> @@ -341,6 +357,12 @@ struct vdpa_config_ops {
>                          u64 iova, u64 size);
>         int (*set_group_asid)(struct vdpa_device *vdev, unsigned int group,
>                               unsigned int asid);
> +       /* IOMMUFD ops */
> +       int (*bind_iommufd)(struct vdpa_device *vdev, struct iommufd_ctx *ictx,
> +                           u32 *out_device_id);
> +       void (*unbind_iommufd)(struct vdpa_device *vdev);
> +       int (*attach_ioas)(struct vdpa_device *vdev, u32 *pt_id);
> +       int (*detach_ioas)(struct vdpa_device *vdev);
>
>         /* Free device resources */
>         void (*free)(struct vdpa_device *vdev);
> @@ -510,4 +532,14 @@ struct vdpa_mgmt_dev {
>  int vdpa_mgmtdev_register(struct vdpa_mgmt_dev *mdev);
>  void vdpa_mgmtdev_unregister(struct vdpa_mgmt_dev *mdev);
>
> -#endif /* _LINUX_VDPA_H */
> +int vdpa_iommufd_physical_bind(struct vdpa_device *vdpa,
> +                              struct iommufd_ctx *ictx, u32 *out_device_id);
> +void vdpa_iommufd_physical_unbind(struct vdpa_device *vdpa);
> +int vdpa_iommufd_physical_attach_ioas(struct vdpa_device *vdpa, u32 *pt_id);
> +int vdpa_iommufd_emulated_bind(struct vdpa_device *vdpa,
> +                              struct iommufd_ctx *ictx, u32 *out_device_id);
> +void vdpa_iommufd_emulated_unbind(struct vdpa_device *vdpa);
> +int vdpa_iommufd_emulated_attach_ioas(struct vdpa_device *vdpa, u32 *pt_id);
> +int vdpa_iommufd_emulated_detach_ioas(struct vdpa_device *vdpa);
> +
> +#endif
> --
> 2.34.3
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC 3/7] vhost: Add 3 new uapi to support iommufd

[Jason Wang]

On Sun, Sep 24, 2023 at 1:05 AM Cindy Lu <lulu@redhat.com> wrote:
>
> VHOST_VDPA_SET_IOMMU_FD: bind the device to iommufd device
>
> VDPA_DEVICE_ATTACH_IOMMUFD_AS: Attach a vdpa device to an iommufd
> address space specified by IOAS id.
>
> VDPA_DEVICE_DETACH_IOMMUFD_AS: Detach a vdpa device
> from the iommufd address space
>
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> ---
>  drivers/vhost/vdpa.c       | 191 +++++++++++++++++++++++++++++++++++++
>  include/uapi/linux/vhost.h |  71 ++++++++++++++
>  2 files changed, 262 insertions(+)
>
> diff --git a/drivers/vhost/vdpa.c b/drivers/vhost/vdpa.c
> index ec32f785dfde..91da012084e9 100644
> --- a/drivers/vhost/vdpa.c
> +++ b/drivers/vhost/vdpa.c
> @@ -18,6 +18,7 @@
>  #include <linux/mm.h>
>  #include <linux/slab.h>
>  #include <linux/iommu.h>
> +#include <linux/iommufd.h>
>  #include <linux/uuid.h>
>  #include <linux/vdpa.h>
>  #include <linux/nospec.h>
> @@ -25,6 +26,8 @@
>
>  #include "vhost.h"
>
> +MODULE_IMPORT_NS(IOMMUFD);
> +
>  enum {
>         VHOST_VDPA_BACKEND_FEATURES =
>         (1ULL << VHOST_BACKEND_F_IOTLB_MSG_V2) |
> @@ -69,6 +72,15 @@ static void vhost_vdpa_iotlb_unmap(struct vhost_vdpa *v,
>                                    struct vhost_iotlb *iotlb, u64 start,
>                                    u64 last, u32 asid);
>
> +void vhost_vdpa_lockdep_assert_held(struct vdpa_device *vdpa)
> +{
> +       struct vhost_vdpa *v = vdpa_get_drvdata(vdpa);
> +
> +       if (WARN_ON(!v))
> +               return;
> +       lockdep_assert_held(&v->vdev.mutex);
> +}
> +
>  static inline u32 iotlb_to_asid(struct vhost_iotlb *iotlb)
>  {
>         struct vhost_vdpa_as *as = container_of(iotlb, struct
> @@ -497,6 +509,173 @@ static long vhost_vdpa_suspend(struct vhost_vdpa *v)
>
>         return ops->suspend(vdpa);
>  }
> +static long vhost_vdpa_tmp_set_iommufd(struct vhost_vdpa *v, void __user *argp)
> +{
> +       struct device *dma_dev = vdpa_get_dma_dev(v->vdpa);
> +       struct vhost_vdpa_set_iommufd set_iommufd;
> +       struct vdpa_device *vdpa = v->vdpa;
> +       struct iommufd_ctx *ictx;
> +       unsigned long minsz;
> +       u32 pt_id, dev_id;
> +       struct fd f;
> +       long r = 0;
> +       int idx;
> +
> +       minsz = offsetofend(struct vhost_vdpa_set_iommufd, ioas_id);
> +       if (copy_from_user(&set_iommufd, argp, minsz))
> +               return -EFAULT;
> +
> +       if (set_iommufd.group_id >= v->nvqs)
> +               return -ENOBUFS;

Needs to be v->ngroups but as replied below, but I think it should be asid.

> +
> +       idx = array_index_nospec(set_iommufd.group_id, v->nvqs);
> +
> +       /* Unset IOMMUFD */
> +       if (set_iommufd.iommufd < 0) {
> +               if (!test_bit(idx, vdpa->vq_bitmap))
> +                       return -EINVAL;
> +
> +               if (!vdpa->iommufd_ictx || !vdpa->iommufd_device)
> +                       return -EINVAL;
> +               if (atomic_read(&vdpa->iommufd_users)) {
> +                       atomic_dec(&vdpa->iommufd_users);
> +                       return 0;
> +               }
> +               vdpa_iommufd_unbind(v->vdpa);
> +               vdpa->iommufd_device = NULL;
> +               vdpa->iommufd_ictx = NULL;
> +               clear_bit(idx, vdpa->vq_bitmap);
> +               return iommu_attach_device(v->domain, dma_dev);
> +       }
> +       /* First opened virtqueue of this vdpa device */
> +
> +       if (!vdpa->vq_bitmap) {
> +               vdpa->vq_bitmap = bitmap_alloc(v->nvqs, GFP_KERNEL);
> +       }
> +       ///if (test_bit(idx, vdpa->vq_bitmap))
> +       //return -EBUSY;
> +
> +       /* For same device but different groups, ++refcount only */
> +       if (vdpa->iommufd_device)
> +               goto out_inc;
> +
> +       r = -EBADF;
> +       f = fdget(set_iommufd.iommufd);
> +       if (!f.file)
> +               goto out_bitmap_free;
> +
> +       r = -EINVAL;
> +       ictx = iommufd_ctx_from_file(f.file);
> +       if (IS_ERR(ictx))
> +               goto out_fdput;
> +
> +       if (v->domain)
> +               iommu_detach_device(v->domain, dma_dev);
> +
> +#if 0
> +       iommu_group_add_device(iommu_group_alloc(), &vdpa->dev);
> +#endif
> +       pt_id = set_iommufd.ioas_id;
> +       r = vdpa_iommufd_bind(vdpa, ictx, &pt_id, &dev_id);
> +       if (r)
> +               goto out_reattach;
> +
> +       set_iommufd.out_dev_id = dev_id;
> +       set_iommufd.out_hwpt_id = pt_id;
> +       r = copy_to_user(argp + minsz, &set_iommufd.out_dev_id,
> +                        sizeof(set_iommufd.out_dev_id) +
> +                                sizeof(set_iommufd.out_hwpt_id)) ?
> +                   -EFAULT :
> +                   0;
> +       if (r)
> +               goto out_device_unbind;
> +       printk(KERN_ERR "[%s] %d called %p\n", __func__, __LINE__,
> +              vdpa->iommufd_ictx);
> +
> +       vdpa->iommufd_ictx = ictx;
> +
> +out_inc:
> +       atomic_inc(&vdpa->iommufd_users);
> +       set_bit(idx, vdpa->vq_bitmap);
> +
> +       goto out_fdput;
> +
> +out_device_unbind:
> +
> +       vdpa_iommufd_unbind(vdpa);
> +out_reattach:
> +
> +       iommu_attach_device(v->domain, dma_dev);
> +       iommufd_ctx_put(ictx);
> +out_fdput:
> +       fdput(f);
> +out_bitmap_free:
> +       bitmap_free(vdpa->vq_bitmap);
> +       return r;
> +}
> +int vdpa_ioctl_device_attach(struct vhost_vdpa *v, void __user *arg)
> +{
> +       struct vdpa_device_attach_iommufd_as attach;
> +       unsigned long minsz;
> +       int ret;
> +
> +       minsz = offsetofend(struct vdpa_device_attach_iommufd_as, ioas_id);
> +
> +       if (copy_from_user(&attach, (void __user *)arg, minsz))
> +               return -EFAULT;
> +
> +       if (attach.argsz < minsz || attach.flags ||
> +           attach.ioas_id == IOMMUFD_INVALID_ID)
> +               return -EINVAL;
> +
> +       if (!v->vdpa->config->bind_iommufd)
> +               return -ENODEV;
> +
> +       if (!v->vdpa->iommufd_ictx) {
> +               ret = -EINVAL;
> +               return ret;
> +       }
> +
> +       ret = v->vdpa->config->attach_ioas(v->vdpa, &attach.ioas_id);
> +
> +       if (ret)
> +               return ret;
> +
> +       ret = copy_to_user(
> +                     (void __user *)arg +
> +                             offsetofend(struct vdpa_device_attach_iommufd_as,
> +                                         flags),
> +                     &attach.ioas_id, sizeof(attach.ioas_id)) ?
> +                     -EFAULT :
> +                     0;
> +
> +       if (ret)
> +               return ret;
> +
> +       return 0;
> +}
> +
> +int vdpa_ioctl_device_detach(struct vhost_vdpa *v, void __user *arg)
> +{
> +       struct vdpa_device_detach_iommufd_as detach;
> +       unsigned long minsz;
> +
> +       minsz = offsetofend(struct vdpa_device_detach_iommufd_as, flags);
> +
> +       if (copy_from_user(&detach, (void __user *)arg, minsz))
> +               return -EFAULT;
> +
> +       if (detach.argsz < minsz || detach.flags)
> +               return -EINVAL;
> +
> +       if (!v->vdpa->config->bind_iommufd)
> +               return -ENODEV;
> +
> +       if (v->vdpa->iommufd_ictx) {
> +               return -EINVAL;
> +       }
> +       return v->vdpa->config->detach_ioas(v->vdpa);
> +}
>
>  static long vhost_vdpa_vring_ioctl(struct vhost_vdpa *v, unsigned int cmd,
>                                    void __user *argp)
> @@ -655,6 +834,18 @@ static long vhost_vdpa_unlocked_ioctl(struct file *filep,
>         case VHOST_SET_LOG_FD:
>                 r = -ENOIOCTLCMD;
>                 break;
> +       case VHOST_VDPA_SET_IOMMU_FD:
> +
> +               r = vhost_vdpa_tmp_set_iommufd(v, argp);
> +               break;
> +       case VDPA_DEVICE_ATTACH_IOMMUFD_AS:
> +               r = vdpa_ioctl_device_attach(v, (void __user *)arg);
> +               break;
> +
> +       case VDPA_DEVICE_DETACH_IOMMUFD_AS:
> +               r = vdpa_ioctl_device_detach(v, (void __user *)arg);
> +               break;
> +
>         case VHOST_VDPA_SET_CONFIG_CALL:
>                 r = vhost_vdpa_set_config_call(v, argp);
>                 break;
> diff --git a/include/uapi/linux/vhost.h b/include/uapi/linux/vhost.h
> index f9f115a7c75b..cdda0c1860d8 100644
> --- a/include/uapi/linux/vhost.h
> +++ b/include/uapi/linux/vhost.h
> @@ -45,6 +45,7 @@
>  #define VHOST_SET_LOG_BASE _IOW(VHOST_VIRTIO, 0x04, __u64)
>  /* Specify an eventfd file descriptor to signal on log write. */
>  #define VHOST_SET_LOG_FD _IOW(VHOST_VIRTIO, 0x07, int)
> +/* Specify an iommufd file descriptor for IO address translation */
>
>  /* Ring setup. */
>  /* Set number of descriptors in ring. This parameter can not
> @@ -180,4 +181,74 @@
>   */
>  #define VHOST_VDPA_SUSPEND             _IO(VHOST_VIRTIO, 0x7D)
>
> +/* vhost vdpa set iommufd
> + * Input parameters:
> + * @iommufd: file descriptor from /dev/iommu; pass -1 to unset
> + * @group_id: identifier of the group that a virtqueue belongs to
> + * @ioas_id: IOAS identifier returned from ioctl(IOMMU_IOAS_ALLOC)
> + * Output parameters:
> + * @out_dev_id: device identifier
> + * @out_hwpt_id: hardware IO pagetable identifier
> + */
> +struct vhost_vdpa_set_iommufd {
> +       __s32 iommufd;
> +       __u32 group_id;

This needs to be ASID of a vdpa device, as the current address space
could be shared among various groups. And the device do translation
based on:

virtqueue -> group -> AS -> IOAS of iommufd

Thanks



> +       __u32 ioas_id;
> +       __u32 out_dev_id;
> +       __u32 out_hwpt_id;
> +};
> +
> +#define VHOST_VDPA_SET_IOMMU_FD \
> +       _IOW(VHOST_VIRTIO, 0x7e, struct vhost_vdpa_set_iommufd)
> +
> +/*
> + * VDPA_DEVICE_ATTACH_IOMMUFD_AS -
> + * _IOW(VHOST_VIRTIO, 0x7f, struct vdpa_device_attach_iommufd_as)
> + *
> + * Attach a vdpa device to an iommufd address space specified by IOAS
> + * id.
> + *
> + * Available only after a device has been bound to iommufd via
> + * VHOST_VDPA_SET_IOMMU_FD
> + *
> + * Undo by VDPA_DEVICE_DETACH_IOMMUFD_AS or device fd close.
> + *
> + * @argsz:     user filled size of this data.
> + * @flags:     must be 0.
> + * @ioas_id:   Input the target id which can represent an ioas
> + *             allocated via iommufd subsystem.
> + *
> + * Return: 0 on success, -errno on failure.
> + */
> +struct vdpa_device_attach_iommufd_as {
> +       __u32 argsz;
> +       __u32 flags;
> +       __u32 ioas_id;
> +};
> +
> +#define VDPA_DEVICE_ATTACH_IOMMUFD_AS \
> +       _IOW(VHOST_VIRTIO, 0x7f, struct vdpa_device_attach_iommufd_as)
> +
> +/*
> + * VDPA_DEVICE_DETACH_IOMMUFD_AS
> + *
> + * Detach a vdpa device from the iommufd address space it has been
> + * attached to. After it, device should be in a blocking DMA state.
> + *
> + * Available only after a device has been bound to iommufd via
> + * VHOST_VDPA_SET_IOMMU_FD
> + *
> + * @argsz:     user filled size of this data.
> + * @flags:     must be 0.
> + *
> + * Return: 0 on success, -errno on failure.
> + */
> +struct vdpa_device_detach_iommufd_as {
> +       __u32 argsz;
> +       __u32 flags;
> +};
> +
> +#define VDPA_DEVICE_DETACH_IOMMUFD_AS \
> +       _IOW(VHOST_VIRTIO, 0x83, struct vdpa_device_detach_iommufd_as)
> +
>  #endif
> --
> 2.34.3
>

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC 1/7] vhost/iommufd: Add the functions support iommufd

[Jason Wang]

On Sun, Sep 24, 2023 at 1:05 AM Cindy Lu <lulu@redhat.com> wrote:
>
> Add a new file vhost/iommufd.c to support the function of
> iommufd, This file contains iommufd function of emulated device and
> the physical device.
>
> Signed-off-by: Cindy Lu <lulu@redhat.com>
> ---
>  drivers/vhost/iommufd.c | 151 ++++++++++++++++++++++++++++++++++++++++
>  drivers/vhost/vhost.h   |  21 ++++++
>  2 files changed, 172 insertions(+)
>  create mode 100644 drivers/vhost/iommufd.c
>
> diff --git a/drivers/vhost/iommufd.c b/drivers/vhost/iommufd.c
> new file mode 100644
> index 000000000000..080858f76fd5
> --- /dev/null
> +++ b/drivers/vhost/iommufd.c
> @@ -0,0 +1,151 @@
> +#include <linux/vdpa.h>
> +#include <linux/iommufd.h>
> +
> +#include "vhost.h"
> +
> +MODULE_IMPORT_NS(IOMMUFD);
> +
> +int vdpa_iommufd_bind(struct vdpa_device *vdpa, struct iommufd_ctx *ictx,
> +                     u32 *ioas_id, u32 *device_id)
> +{
> +       int ret;
> +
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       /*
> +        * If the driver doesn't provide this op then it means the device does
> +        * not do DMA at all. So nothing to do.
> +        */
> +       if (!vdpa->config->bind_iommufd)
> +               return 0;
> +
> +       ret = vdpa->config->bind_iommufd(vdpa, ictx, device_id);
> +       if (ret)
> +               return ret;
> +
> +       ret = vdpa->config->attach_ioas(vdpa, ioas_id);
> +       if (ret)
> +               goto err_unbind;
> +       vdpa->iommufd_attached = true;
> +
> +       return 0;
> +
> +err_unbind:
> +       if (vdpa->config->unbind_iommufd)
> +               vdpa->config->unbind_iommufd(vdpa);
> +       return ret;
> +}
> +
> +void vdpa_iommufd_unbind(struct vdpa_device *vdpa)
> +{
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       if (vdpa->config->unbind_iommufd)
> +               vdpa->config->unbind_iommufd(vdpa);
> +}
> +
> +int vdpa_iommufd_physical_bind(struct vdpa_device *vdpa,
> +                              struct iommufd_ctx *ictx, u32 *out_device_id)
> +{
> +       struct device *dma_dev = vdpa_get_dma_dev(vdpa);
> +       struct iommufd_device *idev;
> +
> +       idev = iommufd_device_bind(ictx, dma_dev, out_device_id);
> +       if (IS_ERR(idev))
> +               return PTR_ERR(idev);
> +       vdpa->iommufd_device = idev;
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(vdpa_iommufd_physical_bind);
> +
> +void vdpa_iommufd_physical_unbind(struct vdpa_device *vdpa)
> +{
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       if (vdpa->iommufd_attached) {
> +               iommufd_device_detach(vdpa->iommufd_device);
> +               vdpa->iommufd_attached = false;
> +       }
> +       iommufd_device_unbind(vdpa->iommufd_device);
> +       vdpa->iommufd_device = NULL;
> +}
> +EXPORT_SYMBOL_GPL(vdpa_iommufd_physical_unbind);
> +
> +int vdpa_iommufd_physical_attach_ioas(struct vdpa_device *vdpa, u32 *pt_id)
> +{
> +       unsigned int flags = 0;
> +
> +       return iommufd_device_attach(vdpa->iommufd_device, pt_id);
> +}
> +EXPORT_SYMBOL_GPL(vdpa_iommufd_physical_attach_ioas);
> +
> +static void vdpa_emulated_unmap(void *data, unsigned long iova,
> +                               unsigned long length)
> +{
> +       struct vdpa_device *vdpa = data;
> +
> +       vdpa->config->dma_unmap(vdpa, 0, iova, length);
> +}
> +
> +static const struct iommufd_access_ops vdpa_user_ops = {
> +       .needs_pin_pages = 1,

Note that simulators support VA, so no page pinning in that case while rebasing.

static bool use_va = true;
module_param(use_va, bool, 0444);
MODULE_PARM_DESC(use_va, "Enable/disable the device's ability to use VA");

So we need to handle that case as well.

(Note that it looks like VA mode is broken, I may need some time to fix that).

> +       .unmap = vdpa_emulated_unmap,
> +};
> +
> +int vdpa_iommufd_emulated_bind(struct vdpa_device *vdpa,
> +                              struct iommufd_ctx *ictx, u32 *out_device_id)
> +{
> +       vhost_vdpa_lockdep_assert_held(vdpa);
> +
> +       vdpa->iommufd_ictx = ictx;
> +       iommufd_ctx_get(ictx);
> +       struct iommufd_device *idev;
> +
> +       idev = iommufd_device_bind(ictx, vdpa->dma_dev, out_device_id);

This seems not appropriate for emulated devices as it deals with the
concepts that only exist in physical devices like the IOMMU domain
etc.

If possible, please refer how VFIO handles this (I guess it should
have something)

Thanks

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC 0/7] vdpa: Add support for iommufd

[Michael S. Tsirkin]

On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> Hi All
> Really apologize for the delay, this is the draft RFC for
> iommufd support for vdpa, This code provides the basic function 
> for iommufd support 
> 
> The code was tested and passed in device vdpa_sim_net
> The qemu code is
> https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> The kernel code is
> https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> 
> ToDo
> 1. this code is out of date and needs to clean and rebase on the latest code 
> 2. this code has some workaround, I Skip the check for
> iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> mutex for iommfd operations 
> 3. only test in emulated device, other modes not tested yet
> 
> After addressed these problems I will send out a new version for RFC. I will
> provide the code in 3 weeks

What's the status here?

-- 
MST

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC 0/7] vdpa: Add support for iommufd

[Michael S. Tsirkin]

On Thu, Oct 26, 2023 at 02:48:07PM +0800, Cindy Lu wrote:
> On Thu, Oct 26, 2023 at 2:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> > > Hi All
> > > Really apologize for the delay, this is the draft RFC for
> > > iommufd support for vdpa, This code provides the basic function
> > > for iommufd support
> > >
> > > The code was tested and passed in device vdpa_sim_net
> > > The qemu code is
> > > https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> > > The kernel code is
> > > https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> > >
> > > ToDo
> > > 1. this code is out of date and needs to clean and rebase on the latest code
> > > 2. this code has some workaround, I Skip the check for
> > > iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> > > mutex for iommfd operations
> > > 3. only test in emulated device, other modes not tested yet
> > >
> > > After addressed these problems I will send out a new version for RFC. I will
> > > provide the code in 3 weeks
> >
> > What's the status here?
> >
> Hi Michael
> The code is finished, but I found some bug after adding the support for ASID,
> will post the new version after this bug is fixed, should be next week
> Thanks
> Cindy


We'll miss this merge window then.

> > --
> > MST
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization

Re: [RFC 0/7] vdpa: Add support for iommufd

[Michael S. Tsirkin]

On Thu, Oct 26, 2023 at 02:48:07PM +0800, Cindy Lu wrote:
> On Thu, Oct 26, 2023 at 2:42 PM Michael S. Tsirkin <mst@redhat.com> wrote:
> >
> > On Sun, Sep 24, 2023 at 01:05:33AM +0800, Cindy Lu wrote:
> > > Hi All
> > > Really apologize for the delay, this is the draft RFC for
> > > iommufd support for vdpa, This code provides the basic function
> > > for iommufd support
> > >
> > > The code was tested and passed in device vdpa_sim_net
> > > The qemu code is
> > > https://gitlab.com/lulu6/gitlabqemutmp/-/tree/iommufdRFC
> > > The kernel code is
> > > https://gitlab.com/lulu6/vhost/-/tree/iommufdRFC
> > >
> > > ToDo
> > > 1. this code is out of date and needs to clean and rebase on the latest code
> > > 2. this code has some workaround, I Skip the check for
> > > iommu_group and CACHE_COHERENCY, also some misc issues like need to add
> > > mutex for iommfd operations
> > > 3. only test in emulated device, other modes not tested yet
> > >
> > > After addressed these problems I will send out a new version for RFC. I will
> > > provide the code in 3 weeks
> >
> > What's the status here?
> >
> Hi Michael
> The code is finished, but I found some bug after adding the support for ASID,
> will post the new version after this bug is fixed, should be next week
> Thanks
> Cindy

The week is almost gone, what's going on?


> > --
> > MST
> >

_______________________________________________
Virtualization mailing list
Virtualization@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/virtualization