[PATCH] kvm: lock slots_lock around device assignment

[PATCH] kvm: lock slots_lock around device assignment

[Alex Williamson]

As pointed out by Jason Baron, when assigning a device to a guest
we first set the iommu domain pointer, which enables mapping
and unmapping of memory slots to the iommu.  This leaves a window
where this path is enabled, but we haven't synchronized the iommu
mappings to the existing memory slots.  Thus a slot being removed
at that point could send us down unexpected code paths removing
non-existent pinnings and iommu mappings.  Take the slots_lock
around creating the iommu domain and initial mappings as well as
around iommu teardown to avoid this race.

Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
---

 virt/kvm/iommu.c |   23 +++++++++++++++--------
 1 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/virt/kvm/iommu.c b/virt/kvm/iommu.c
index fec1723..e9fff98 100644
--- a/virt/kvm/iommu.c
+++ b/virt/kvm/iommu.c
@@ -240,9 +240,13 @@ int kvm_iommu_map_guest(struct kvm *kvm)
 		return -ENODEV;
 	}
 
+	mutex_lock(&kvm->slots_lock);
+
 	kvm->arch.iommu_domain = iommu_domain_alloc(&pci_bus_type);
-	if (!kvm->arch.iommu_domain)
-		return -ENOMEM;
+	if (!kvm->arch.iommu_domain) {
+		r = -ENOMEM;
+		goto out_unlock;
+	}
 
 	if (!allow_unsafe_assigned_interrupts &&
 	    !iommu_domain_has_cap(kvm->arch.iommu_domain,
@@ -253,17 +257,16 @@ int kvm_iommu_map_guest(struct kvm *kvm)
 		       " module option.\n", __func__);
 		iommu_domain_free(kvm->arch.iommu_domain);
 		kvm->arch.iommu_domain = NULL;
-		return -EPERM;
+		r = -EPERM;
+		goto out_unlock;
 	}
 
 	r = kvm_iommu_map_memslots(kvm);
 	if (r)
-		goto out_unmap;
-
-	return 0;
+		kvm_iommu_unmap_memslots(kvm);
 
-out_unmap:
-	kvm_iommu_unmap_memslots(kvm);
+out_unlock:
+	mutex_unlock(&kvm->slots_lock);
 	return r;
 }
 
@@ -340,7 +343,11 @@ int kvm_iommu_unmap_guest(struct kvm *kvm)
 	if (!domain)
 		return 0;
 
+	mutex_lock(&kvm->slots_lock);
 	kvm_iommu_unmap_memslots(kvm);
+	kvm->arch.iommu_domain = NULL;
+	mutex_unlock(&kvm->slots_lock);
+
 	iommu_domain_free(domain);
 	return 0;
 }

Re: [PATCH] kvm: lock slots_lock around device assignment

[Marcelo Tosatti]

On Tue, Apr 17, 2012 at 09:46:44PM -0600, Alex Williamson wrote:
> @@ -340,7 +343,11 @@ int kvm_iommu_unmap_guest(struct kvm *kvm)
>  	if (!domain)
>  		return 0;
>  
> +	mutex_lock(&kvm->slots_lock);
>  	kvm_iommu_unmap_memslots(kvm);
> +	kvm->arch.iommu_domain = NULL;
> +	mutex_unlock(&kvm->slots_lock);
> +
>  	iommu_domain_free(domain);
>  	return 0;
>  }

This might trigger lockdep warnings due to

kvm_vm_ioctl_create_vcpu
mutex_lock(&kvm->lock)
kvm_put_kvm(kvm)
kvm_destroy_vm
kvm_iommu_unmap_guest 

sequence.

Better drop it, it is not necessary in vm destruction
path (since only user is self).

Re: [PATCH] kvm: lock slots_lock around device assignment

[Alex Williamson]

On Wed, 2012-04-18 at 23:30 -0300, Marcelo Tosatti wrote:
> On Tue, Apr 17, 2012 at 09:46:44PM -0600, Alex Williamson wrote:
> > @@ -340,7 +343,11 @@ int kvm_iommu_unmap_guest(struct kvm *kvm)
> >  	if (!domain)
> >  		return 0;
> >  
> > +	mutex_lock(&kvm->slots_lock);
> >  	kvm_iommu_unmap_memslots(kvm);
> > +	kvm->arch.iommu_domain = NULL;
> > +	mutex_unlock(&kvm->slots_lock);
> > +
> >  	iommu_domain_free(domain);
> >  	return 0;
> >  }
> 
> This might trigger lockdep warnings due to
> 
> kvm_vm_ioctl_create_vcpu
> mutex_lock(&kvm->lock)
> kvm_put_kvm(kvm)
> kvm_destroy_vm
> kvm_iommu_unmap_guest 
> 
> sequence.
> 
> Better drop it, it is not necessary in vm destruction
> path (since only user is self).

I actually ran this with lockdep and didn't generate a warning;
hopefully I had it configured correctly.  Also, we'll soon be unmapping
the guest any time we remove the last assigned device so this will no
longer be a vm destruction-only path.  We can just as easily race adding
new mappings or removing already removed ones on that path.  We also
acquire kvm->lock in the mapping path:

kvm_vm_ioctl_assign_device() {
mutex_lock(&kvm->lock);
if (!kvm->arch.iommu_domain) {
      r = kvm_iommu_map_guest(kvm);

which by inspection and the lock ordering note in kvm_main seems to be
ok.  Thanks,

Alex

Re: [PATCH] kvm: lock slots_lock around device assignment

[Marcelo Tosatti]

On Tue, Apr 17, 2012 at 09:46:44PM -0600, Alex Williamson wrote:
> As pointed out by Jason Baron, when assigning a device to a guest
> we first set the iommu domain pointer, which enables mapping
> and unmapping of memory slots to the iommu.  This leaves a window
> where this path is enabled, but we haven't synchronized the iommu
> mappings to the existing memory slots.  Thus a slot being removed
> at that point could send us down unexpected code paths removing
> non-existent pinnings and iommu mappings.  Take the slots_lock
> around creating the iommu domain and initial mappings as well as
> around iommu teardown to avoid this race.
> 
> Signed-off-by: Alex Williamson <alex.williamson@redhat.com>

Applied, thanks.