Re: [v6 2/3] vfio: support notifier chain in vfio_group

[Jike Song <jike.song@intel.com>]

On 11/22/2016 12:56 AM, Alex Williamson wrote:
> On Sat, 19 Nov 2016 12:14:29 +0800
> Jike Song <jike.song@intel.com> wrote:
> 
>> On 11/19/2016 01:55 AM, Alex Williamson wrote:
>>> On Fri, 18 Nov 2016 19:01:48 +0800
>>> Jike Song <jike.song@intel.com> wrote:  
>>>> Beyond vfio_iommu events, users might also be interested in
>>>> vfio_group events. For example, if a vfio_group is used along
>>>> with Qemu/KVM, whenever kvm pointer is set to/cleared from the
>>>> vfio_group, users could be notified.
>>>>
>>>> Currently only VFIO_GROUP_NOTIFY_SET_KVM supported.
>>>>
>>>> Cc: Alex Williamson <alex.williamson@redhat.com>
>>>> Cc: Kirti Wankhede <kwankhede@nvidia.com>
>>>> Cc: Paolo Bonzini <pbonzini@redhat.com>
>>>> Cc: Xiao Guangrong <guangrong.xiao@linux.intel.com>
>>>> Signed-off-by: Jike Song <jike.song@intel.com>
>>>> ---
>>>>  drivers/vfio/vfio.c  | 69 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>  include/linux/vfio.h |  6 +++++
>>>>  2 files changed, 75 insertions(+)
>>>>
>>>> diff --git a/drivers/vfio/vfio.c b/drivers/vfio/vfio.c
>>>> index ec62bec..e2bb197 100644
>>>> --- a/drivers/vfio/vfio.c
>>>> +++ b/drivers/vfio/vfio.c
>>>> @@ -86,6 +86,8 @@ struct vfio_group {
>>>>  	struct mutex			unbound_lock;
>>>>  	atomic_t			opened;
>>>>  	bool				noiommu;
>>>> +	struct kvm			*kvm;
>>>> +	struct blocking_notifier_head	notifier;
>>>>  };
>>>>  
>>>>  struct vfio_device {
>>>> @@ -339,6 +341,7 @@ static struct vfio_group *vfio_create_group(struct iommu_group *iommu_group)
>>>>  #ifdef CONFIG_VFIO_NOIOMMU
>>>>  	group->noiommu = (iommu_group_get_iommudata(iommu_group) == &noiommu);
>>>>  #endif
>>>> +	BLOCKING_INIT_NOTIFIER_HEAD(&group->notifier);
>>>>  
>>>>  	group->nb.notifier_call = vfio_iommu_group_notifier;
>>>>  
>>>> @@ -1015,6 +1018,63 @@ static long vfio_ioctl_check_extension(struct vfio_container *container,
>>>>  	return ret;
>>>>  }
>>>>  
>>>> +void vfio_group_set_kvm(struct vfio_group *group, struct kvm *kvm)
>>>> +{
>>>> +	group->kvm = kvm;
>>>> +	blocking_notifier_call_chain(&group->notifier,
>>>> +				VFIO_GROUP_NOTIFY_SET_KVM, kvm);
>>>> +}
>>>> +EXPORT_SYMBOL_GPL(vfio_group_set_kvm);
>>>> +
>>>> +static int vfio_register_group_notifier(struct vfio_group *group,
>>>> +					unsigned long *events,
>>>> +					struct notifier_block *nb)
>>>> +{
>>>> +	int ret;
>>>> +	bool set_kvm = false;
>>>> +
>>>> +	if (*events & VFIO_GROUP_NOTIFY_SET_KVM)
>>>> +		set_kvm = true;
>>>> +
>>>> +	/* clear known events */
>>>> +	*events &= ~VFIO_GROUP_NOTIFY_SET_KVM;
>>>> +
>>>> +	/* refuse to continue if still events remaining */
>>>> +	if (*events)
>>>> +		return -EINVAL;
>>>> +
>>>> +	if (!atomic_inc_not_zero(&group->opened))
>>>> +		return -EINVAL;  
>>>
>>> vfio_group.opened is only used to make sure we don't allow multiple
>>> opens of the group, incrementing it doesn't currently assure the group
>>> remains opened.  What happens if the user process releases the group in
>>> the midst of this?  
>>
>> Thanks for pointing out this.
>> It seems okay to think the group is open by checking group->opened,
>> but after that I failed to find any existing method to prevent a concurrent
>> vfio_group_fops_release, could you enlighten me a bit?
> 
> I don't think we have such a thing.  I briefly investigated whether we
> should add a group mutex rather than the atomic, but at this point I'm
> just leaning towards using the same conditions as attaching the iommu
> notifier, ie. call vfio_group_add_container_user(). 

Thanks, indeed it seems that a mutex is needed to protect the group itself,
but on the other hand, could open and release of group fops be actually
concurrent? I don't have a clear understanding here but doubt there won't.

> This is also what
> we do for vfio_group_get_external_user() so I think it makes sense that
> any sort of group reference or notifier registration have the same
> requirements.  Thanks,

Cooked a v7 according to your idea, please kindly have a look. Thanks!


--
Thanks,
Jike