From: Alex Williamson <alex.williamson@redhat.com>
To: Jike Song <jike.song@intel.com>
Cc: Kirti Wankhede <kwankhede@nvidia.com>,
pbonzini@redhat.com, kraxel@redhat.com, cjia@nvidia.com,
qemu-devel@nongnu.org, kvm@vger.kernel.org, kevin.tian@intel.com,
bjsdjshi@linux.vnet.ibm.com, linux-kernel@vger.kernel.org
Subject: Re: [PATCH v10 10/19] vfio iommu: Add blocking notifier to notify DMA_UNMAP
Date: Fri, 28 Oct 2016 06:40:45 -0600 [thread overview]
Message-ID: <20161028064045.0e8ca7dc@t450s.home> (raw)
In-Reply-To: <5812FF66.6020801@intel.com>
On Fri, 28 Oct 2016 15:33:58 +0800
Jike Song <jike.song@intel.com> wrote:
> On 10/27/2016 05:29 AM, Kirti Wankhede wrote:
> > Added blocking notifier to IOMMU TYPE1 driver to notify vendor drivers
> > about DMA_UNMAP.
> > Exported two APIs vfio_register_notifier() and vfio_unregister_notifier().
> > Vendor driver should register notifer using these APIs.
> > Vendor driver should use VFIO_IOMMU_NOTIFY_DMA_UNMAP action to invalidate
> > mappings.
> >
> > Signed-off-by: Kirti Wankhede <kwankhede@nvidia.com>
> > Signed-off-by: Neo Jia <cjia@nvidia.com>
> > Change-Id: I5910d0024d6be87f3e8d3e0ca0eaeaaa0b17f271
> > ---
> > drivers/vfio/vfio.c | 73 +++++++++++++++++++++++++++++++++
> > drivers/vfio/vfio_iommu_type1.c | 89 ++++++++++++++++++++++++++++++++++++-----
> > include/linux/vfio.h | 11 +++++
> > 3 files changed, 163 insertions(+), 10 deletions(-)
> >
> > diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
> > index 28b50ca14c52..ff05ac6b1e90 100644
> > --- a/drivers/vfio/vfio.c
> > +++ b/drivers/vfio/vfio.c
> > @@ -1891,6 +1891,79 @@ err_unpin_pages:
> > }
> > EXPORT_SYMBOL(vfio_unpin_pages);
> >
> > +int vfio_register_notifier(struct device *dev, struct notifier_block *nb)
> > +{
> > + struct vfio_container *container;
> > + struct vfio_group *group;
> > + struct vfio_iommu_driver *driver;
> > + ssize_t ret;
> > +
> > + if (!dev || !nb)
> > + return -EINVAL;
> > +
> > + group = vfio_group_get_from_dev(dev);
> > + if (IS_ERR(group))
> > + return PTR_ERR(group);
> > +
> > + ret = vfio_group_add_container_user(group);
> > + if (ret)
> > + goto err_register_nb;
> > +
> > + container = group->container;
> > + down_read(&container->group_lock);
> > +
> > + driver = container->iommu_driver;
> > + if (likely(driver && driver->ops->register_notifier))
> > + ret = driver->ops->register_notifier(container->iommu_data, nb);
> > + else
> > + ret = -EINVAL;
> > +
> > + up_read(&container->group_lock);
> > + vfio_group_try_dissolve_container(group);
> > +
> > +err_register_nb:
> > + vfio_group_put(group);
> > + return ret;
> > +}
> > +EXPORT_SYMBOL(vfio_register_notifier);
> > +
> > +int vfio_unregister_notifier(struct device *dev, struct notifier_block *nb)
> > +{
> > + struct vfio_container *container;
> > + struct vfio_group *group;
> > + struct vfio_iommu_driver *driver;
> > + ssize_t ret;
> > +
> > + if (!dev || !nb)
> > + return -EINVAL;
> > +
> > + group = vfio_group_get_from_dev(dev);
> > + if (IS_ERR(group))
> > + return PTR_ERR(group);
> > +
> > + ret = vfio_group_add_container_user(group);
> > + if (ret)
> > + goto err_unregister_nb;
> > +
> > + container = group->container;
> > + down_read(&container->group_lock);
> > +
> > + driver = container->iommu_driver;
> > + if (likely(driver && driver->ops->unregister_notifier))
> > + ret = driver->ops->unregister_notifier(container->iommu_data,
> > + nb);
> > + else
> > + ret = -EINVAL;
> > +
> > + up_read(&container->group_lock);
> > + vfio_group_try_dissolve_container(group);
> > +
> > +err_unregister_nb:
> > + vfio_group_put(group);
> > + return ret;
> > +}
> > +EXPORT_SYMBOL(vfio_unregister_notifier);
> > +
> > /**
> > * Module/class support
> > */
> > diff --git a/drivers/vfio/vfio_iommu_type1.c b/drivers/vfio/vfio_iommu_type1.c
> > index 5add11a147e1..a4bd331ac0fd 100644
> > --- a/drivers/vfio/vfio_iommu_type1.c
> > +++ b/drivers/vfio/vfio_iommu_type1.c
> > @@ -37,6 +37,7 @@
> > #include <linux/vfio.h>
> > #include <linux/workqueue.h>
> > #include <linux/mdev.h>
> > +#include <linux/notifier.h>
> >
> > #define DRIVER_VERSION "0.2"
> > #define DRIVER_AUTHOR "Alex Williamson <alex.williamson@redhat.com>"
> > @@ -59,6 +60,7 @@ struct vfio_iommu {
> > struct vfio_domain *external_domain; /* domain for external user */
> > struct mutex lock;
> > struct rb_root dma_list;
> > + struct blocking_notifier_head notifier;
> > bool v2;
> > bool nesting;
> > };
> > @@ -549,7 +551,8 @@ static long vfio_iommu_type1_pin_pages(void *iommu_data,
> >
> > mutex_lock(&iommu->lock);
> >
> > - if (!iommu->external_domain) {
> > + /* Fail if notifier list is empty */
> > + if ((!iommu->external_domain) || (!iommu->notifier.head)) {
> > ret = -EINVAL;
> > goto pin_done;
> > }
> > @@ -768,6 +771,50 @@ static unsigned long vfio_pgsize_bitmap(struct vfio_iommu *iommu)
> > return bitmap;
> > }
> >
> > +/*
> > + * This function finds pfn in domain->external_addr_space->pfn_list for given
> > + * iova range. If pfn exist, notify pfn to registered notifier list. On
> > + * receiving notifier callback, vendor driver should invalidate the mapping and
> > + * call vfio_unpin_pages() to unpin this pfn. With that vfio_pfn for this pfn
> > + * gets removed from rb tree of pfn_list. That re-arranges rb tree, so while
> > + * searching for next vfio_pfn in rb tree, start search from first node again.
> > + * If any vendor driver doesn't unpin that pfn, vfio_pfn would not get removed
> > + * from rb tree and so in next search vfio_pfn would be same as previous
> > + * vfio_pfn. In that case, exit from loop.
> > + */
> > +static void vfio_notifier_call_chain(struct vfio_iommu *iommu,
> > + struct vfio_iommu_type1_dma_unmap *unmap)
> > +{
> > + struct vfio_domain *domain = iommu->external_domain;
> > + struct rb_node *n;
> > + struct vfio_pfn *vpfn = NULL, *prev_vpfn;
> > +
> > + do {
> > + prev_vpfn = vpfn;
> > + mutex_lock(&domain->external_addr_space->pfn_list_lock);
> > +
> > + n = rb_first(&domain->external_addr_space->pfn_list);
> > +
> > + for (; n; n = rb_next(n), vpfn = NULL) {
> > + vpfn = rb_entry(n, struct vfio_pfn, node);
> > +
> > + if ((vpfn->iova >= unmap->iova) &&
> > + (vpfn->iova < unmap->iova + unmap->size))
> > + break;
> > + }
> > +
> > + mutex_unlock(&domain->external_addr_space->pfn_list_lock);
> > +
> > + /* Notify any listeners about DMA_UNMAP */
> > + if (vpfn)
> > + blocking_notifier_call_chain(&iommu->notifier,
> > + VFIO_IOMMU_NOTIFY_DMA_UNMAP,
> > + &vpfn->pfn);
>
> Hi Kirti,
>
> The information carried by notifier is only a pfn.
>
> Since your pin/unpin interfaces design, it's the vendor driver who should
> guarantee pin/unpin same times. To achieve that, the vendor driver must
> cache it's iova->pfn mapping on its side, to avoid pinning a same page
> for multiple times.
>
> With the notifier carrying only a pfn, to find the iova by this pfn,
> the vendor driver must *also* keep a reverse-mapping. That's a bit
> too much.
>
> Since the vendor could also suffer from IOMMU-compatible problem,
> which means a local cache is always helpful, so I'd like to have the
> iova carried to the notifier.
>
> What'd you say?
I agree, the pfn is not unique, multiple guest pfns (iovas) might be
backed by the same host pfn. DMA_UNMAP calls are based on iova, the
notifier through to the vendor driver must be based on the same.
Thanks,
Alex
> > + } while (vpfn && (prev_vpfn != vpfn));
> > +
> > + WARN_ON(vpfn);
> > +}
> > +
> > static int vfio_dma_do_unmap(struct vfio_iommu *iommu,
> > struct vfio_iommu_type1_dma_unmap *unmap)
> > {
> > @@ -844,6 +891,9 @@ unlock:
> > /* Report how much was unmapped */
> > unmap->size = unmapped;
> >
> > + if (unmapped && iommu->external_domain)
> > + vfio_notifier_call_chain(iommu, unmap);
> > +
> > return ret;
> > }
> >
> > @@ -1418,6 +1468,7 @@ static void *vfio_iommu_type1_open(unsigned long arg)
> > INIT_LIST_HEAD(&iommu->domain_list);
> > iommu->dma_list = RB_ROOT;
> > mutex_init(&iommu->lock);
> > + BLOCKING_INIT_NOTIFIER_HEAD(&iommu->notifier);
> >
> > return iommu;
> > }
> > @@ -1555,16 +1606,34 @@ static long vfio_iommu_type1_ioctl(void *iommu_data,
> > return -ENOTTY;
> > }
> >
> > +static int vfio_iommu_type1_register_notifier(void *iommu_data,
> > + struct notifier_block *nb)
> > +{
> > + struct vfio_iommu *iommu = iommu_data;
> > +
> > + return blocking_notifier_chain_register(&iommu->notifier, nb);
> > +}
> > +
> > +static int vfio_iommu_type1_unregister_notifier(void *iommu_data,
> > + struct notifier_block *nb)
> > +{
> > + struct vfio_iommu *iommu = iommu_data;
> > +
> > + return blocking_notifier_chain_unregister(&iommu->notifier, nb);
> > +}
> > +
> > static const struct vfio_iommu_driver_ops vfio_iommu_driver_ops_type1 = {
> > - .name = "vfio-iommu-type1",
> > - .owner = THIS_MODULE,
> > - .open = vfio_iommu_type1_open,
> > - .release = vfio_iommu_type1_release,
> > - .ioctl = vfio_iommu_type1_ioctl,
> > - .attach_group = vfio_iommu_type1_attach_group,
> > - .detach_group = vfio_iommu_type1_detach_group,
> > - .pin_pages = vfio_iommu_type1_pin_pages,
> > - .unpin_pages = vfio_iommu_type1_unpin_pages,
> > + .name = "vfio-iommu-type1",
> > + .owner = THIS_MODULE,
> > + .open = vfio_iommu_type1_open,
> > + .release = vfio_iommu_type1_release,
> > + .ioctl = vfio_iommu_type1_ioctl,
> > + .attach_group = vfio_iommu_type1_attach_group,
> > + .detach_group = vfio_iommu_type1_detach_group,
> > + .pin_pages = vfio_iommu_type1_pin_pages,
> > + .unpin_pages = vfio_iommu_type1_unpin_pages,
> > + .register_notifier = vfio_iommu_type1_register_notifier,
> > + .unregister_notifier = vfio_iommu_type1_unregister_notifier,
> > };
> >
> > static int __init vfio_iommu_type1_init(void)
> > diff --git a/include/linux/vfio.h b/include/linux/vfio.h
> > index 0609a2052846..4c91ce8bfaeb 100644
> > --- a/include/linux/vfio.h
> > +++ b/include/linux/vfio.h
> > @@ -80,6 +80,10 @@ struct vfio_iommu_driver_ops {
> > unsigned long *phys_pfn);
> > long (*unpin_pages)(void *iommu_data, unsigned long *pfn,
> > long npage);
> > + int (*register_notifier)(void *iommu_data,
> > + struct notifier_block *nb);
> > + int (*unregister_notifier)(void *iommu_data,
> > + struct notifier_block *nb);
> > };
> >
> > extern int vfio_register_iommu_driver(const struct vfio_iommu_driver_ops *ops);
> > @@ -137,6 +141,13 @@ extern long vfio_pin_pages(struct device *dev, unsigned long *user_pfn,
> > extern long vfio_unpin_pages(struct device *dev, unsigned long *pfn,
> > long npage);
> >
> > +#define VFIO_IOMMU_NOTIFY_DMA_UNMAP 1
> > +
> > +extern int vfio_register_notifier(struct device *dev,
> > + struct notifier_block *nb);
> > +
> > +extern int vfio_unregister_notifier(struct device *dev,
> > + struct notifier_block *nb);
> > /*
> > * IRQfd - generic
> > */
> >
>
next prev parent reply other threads:[~2016-10-28 12:40 UTC|newest]
Thread overview: 56+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-10-26 21:29 [PATCH v10 00/19] Add Mediated device support Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 01/19] vfio: Mediated device Core driver Kirti Wankhede
2016-10-29 4:30 ` Jike Song
2016-10-29 10:06 ` Kirti Wankhede
2016-10-29 18:11 ` Jike Song
2016-11-02 7:59 ` Kirti Wankhede
2016-11-02 10:31 ` [Qemu-devel] " Jike Song
2016-11-01 3:08 ` Jike Song
2016-11-01 3:44 ` Alex Williamson
2016-11-01 5:28 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 02/19] vfio: VFIO based driver for Mediated devices Kirti Wankhede
2016-11-02 10:39 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 03/19] vfio: Rearrange functions to get vfio_group from dev Kirti Wankhede
2016-11-02 10:41 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 04/19] vfio: Common function to increment container_users Kirti Wankhede
2016-11-02 11:34 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 05/19] vfio iommu: Added pin and unpin callback functions to vfio_iommu_driver_ops Kirti Wankhede
2016-11-01 8:07 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 06/19] vfio iommu type1: Update arguments of vfio_lock_acct Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 07/19] vfio iommu type1: Update argument of vaddr_get_pfn() Kirti Wankhede
2016-10-27 12:11 ` Jike Song
2016-10-27 12:24 ` Kirti Wankhede
2016-10-28 6:01 ` Jike Song
2016-11-02 8:06 ` Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 08/19] vfio iommu type1: Add find_iommu_group() function Kirti Wankhede
2016-11-02 14:13 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 09/19] vfio iommu type1: Add support for mediated devices Kirti Wankhede
2016-10-27 23:01 ` Alex Williamson
2016-11-02 13:29 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 10/19] vfio iommu: Add blocking notifier to notify DMA_UNMAP Kirti Wankhede
2016-10-28 7:33 ` Jike Song
2016-10-28 12:38 ` Kirti Wankhede
2016-10-28 12:40 ` Alex Williamson [this message]
2016-10-28 20:02 ` Kirti Wankhede
2016-10-28 20:33 ` Alex Williamson
2016-10-29 10:37 ` Kirti Wankhede
2016-10-29 14:03 ` Alex Williamson
[not found] ` <20161101034558.GA7186@bjsdjshi@linux.vnet.ibm.com>
2016-11-01 7:47 ` Kirti Wankhede
2016-10-31 3:50 ` Jike Song
2016-10-31 5:59 ` Kirti Wankhede
2016-10-31 6:05 ` Jike Song
2016-10-26 21:29 ` [PATCH v10 11/19] vfio: Introduce common function to add capabilities Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 12/19] vfio_pci: Update vfio_pci to use vfio_info_add_capability() Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 13/19] vfio: Introduce vfio_set_irqs_validate_and_prepare() Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 14/19] vfio_pci: Updated to use vfio_set_irqs_validate_and_prepare() Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 15/19] vfio_platform: " Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 16/19] vfio: Define device_api strings Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 17/19] docs: Add Documentation for Mediated devices Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 18/19] docs: Sysfs ABI for mediated device framework Kirti Wankhede
2016-10-31 7:19 ` Jike Song
2016-11-02 7:55 ` Kirti Wankhede
2016-10-26 21:29 ` [PATCH v10 19/19] docs: Sample driver to demonstrate how to use Mediated " Kirti Wankhede
2016-10-27 14:29 ` Jonathan Corbet
2016-11-01 8:32 ` [PATCH v10 00/19] Add Mediated device support Jike Song
2016-11-01 15:24 ` Gerd Hoffmann
2016-11-02 1:01 ` [Qemu-devel] " Jike Song
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20161028064045.0e8ca7dc@t450s.home \
--to=alex.williamson@redhat.com \
--cc=bjsdjshi@linux.vnet.ibm.com \
--cc=cjia@nvidia.com \
--cc=jike.song@intel.com \
--cc=kevin.tian@intel.com \
--cc=kraxel@redhat.com \
--cc=kvm@vger.kernel.org \
--cc=kwankhede@nvidia.com \
--cc=linux-kernel@vger.kernel.org \
--cc=pbonzini@redhat.com \
--cc=qemu-devel@nongnu.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).