[PATCH] iommu/vt-d: Fix VM domain ID leak

[PATCH] iommu/vt-d: Fix VM domain ID leak

[Alex Williamson]

This continues the attempt to fix commit fb170fb4c548 ("iommu/vt-d:
Introduce helper functions to make code symmetric for readability").
The previous attempt in commit 71684406905f ("iommu/vt-d: Detach
domain *only* from attached iommus") overlooked the fact that
dmar_domain.iommu_bmp gets cleared for VM domains when devices are
detached:

intel_iommu_detach_device
  domain_remove_one_dev_info
    domain_detach_iommu

The domain is detached from the iommu, but the iommu is still attached
to the domain, for whatever reason.  Thus when we get to domain_exit(),
we can't rely on iommu_bmp for VM domains to find the active iommus,
we must check them all.  Without that, the corresponding bit in
intel_iommu.domain_ids doesn't get cleared and repeated VM domain
creation and destruction will run out of domain IDs.  Meanwhile we
still can't call iommu_detach_domain() on arbitrary non-VM domains or
we risk clearing in-use domain IDs, as 71684406905f attempted to
address.

It's tempting to modify iommu_detach_domain() to test the domain
iommu_bmp, but the call ordering from domain_remove_one_dev_info()
prevents it being able to work as fb170fb4c548 seems to have intended.
Caching of unused VM domains on the iommu object seems to be the root
of the problem, but this code is far too fragile for that kind of
rework to be proposed for stable, so we simply revert this chunk to
its state prior to fb170fb4c548.

Fixes: fb170fb4c548 ("iommu/vt-d: Introduce helper functions to make
                      code symmetric for readability")
Fixes: 71684406905f ("iommu/vt-d: Detach domain *only* from attached
                      iommus")
Signed-off-by: Alex Williamson <alex.williamson@redhat.com>
Cc: Jiang Liu <jiang.liu@linux.intel.com>
Cc: stable@vger.kernel.org # v3.17+
---
 drivers/iommu/intel-iommu.c |    9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index a98a7b2..0649b94 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -1830,8 +1830,9 @@ static int domain_init(struct dmar_domain *domain, int guest_width)
 
 static void domain_exit(struct dmar_domain *domain)
 {
+	struct dmar_drhd_unit *drhd;
+	struct intel_iommu *iommu;
 	struct page *freelist = NULL;
-	int i;
 
 	/* Domain 0 is reserved, so dont process it */
 	if (!domain)
@@ -1851,8 +1852,10 @@ static void domain_exit(struct dmar_domain *domain)
 
 	/* clear attached or cached domains */
 	rcu_read_lock();
-	for_each_set_bit(i, domain->iommu_bmp, g_num_of_iommus)
-		iommu_detach_domain(domain, g_iommus[i]);
+	for_each_active_iommu(iommu, drhd)
+		if (domain_type_is_vm(domain) ||
+		    test_bit(iommu->seq_id, domain->iommu_bmp))
+			iommu_detach_domain(domain, iommu);
 	rcu_read_unlock();
 
 	dma_free_pagelist(freelist);

Re: [PATCH] iommu/vt-d: Fix VM domain ID leak

[Joerg Roedel]

On Tue, Jul 14, 2015 at 02:48:53PM -0600, Alex Williamson wrote:
> This continues the attempt to fix commit fb170fb4c548 ("iommu/vt-d:
> Introduce helper functions to make code symmetric for readability").
> The previous attempt in commit 71684406905f ("iommu/vt-d: Detach
> domain *only* from attached iommus") overlooked the fact that
> dmar_domain.iommu_bmp gets cleared for VM domains when devices are
> detached:
> 
> intel_iommu_detach_device
>   domain_remove_one_dev_info
>     domain_detach_iommu
> 
> The domain is detached from the iommu, but the iommu is still attached
> to the domain, for whatever reason.

Gaah, this whole vm_or_si special handling is a complete and fragile
mess.

The only reason for keeping the domain-id allocated in the iommu
seems to be that it can be re-used later, when another device behind
this iommu is attached to the domain. But this is hardly a justification
for the complexity and special case handling here, so how about this
diff instead:

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index a98a7b2..c8fc8c8 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -4618,8 +4618,7 @@ static void domain_remove_one_dev_info(struct dmar_domain *domain,
 
 	if (found == 0) {
 		domain_detach_iommu(domain, iommu);
-		if (!domain_type_is_vm_or_si(domain))
-			iommu_detach_domain(domain, iommu);
+		iommu_detach_domain(domain, iommu);
 	}
 }
 
This removes the caching of domain-ids, and iommu_detach_domain
correctly handles all types of domains (dma-api, vm and si). It should
be safe, but can you please double-check?


	Joerg

Re: [PATCH] iommu/vt-d: Fix VM domain ID leak

[Alex Williamson]

On Thu, 2015-07-16 at 17:20 +0200, Joerg Roedel wrote:
> On Tue, Jul 14, 2015 at 02:48:53PM -0600, Alex Williamson wrote:
> > This continues the attempt to fix commit fb170fb4c548 ("iommu/vt-d:
> > Introduce helper functions to make code symmetric for readability").
> > The previous attempt in commit 71684406905f ("iommu/vt-d: Detach
> > domain *only* from attached iommus") overlooked the fact that
> > dmar_domain.iommu_bmp gets cleared for VM domains when devices are
> > detached:
> > 
> > intel_iommu_detach_device
> >   domain_remove_one_dev_info
> >     domain_detach_iommu
> > 
> > The domain is detached from the iommu, but the iommu is still attached
> > to the domain, for whatever reason.
> 
> Gaah, this whole vm_or_si special handling is a complete and fragile
> mess.

+1

> The only reason for keeping the domain-id allocated in the iommu
> seems to be that it can be re-used later, when another device behind
> this iommu is attached to the domain. But this is hardly a justification
> for the complexity and special case handling here, so how about this
> diff instead:
> 
> diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
> index a98a7b2..c8fc8c8 100644
> --- a/drivers/iommu/intel-iommu.c
> +++ b/drivers/iommu/intel-iommu.c
> @@ -4618,8 +4618,7 @@ static void domain_remove_one_dev_info(struct dmar_domain *domain,
>  
>  	if (found == 0) {
>  		domain_detach_iommu(domain, iommu);
> -		if (!domain_type_is_vm_or_si(domain))
> -			iommu_detach_domain(domain, iommu);
> +		iommu_detach_domain(domain, iommu);
>  	}
>  }
>  
> This removes the caching of domain-ids, and iommu_detach_domain
> correctly handles all types of domains (dma-api, vm and si). It should
> be safe, but can you please double-check?

I was tempted to do this as well, but what about
domain_remove_dev_info() where we also handle vm domains specially and
don't do a symmetric detach in both directions?  I started down that
path but quickly found the code to fragile to make those kinds of
changes.  Thanks,

Alex

Re: [PATCH] iommu/vt-d: Fix VM domain ID leak

[Joerg Roedel]

On Thu, Jul 16, 2015 at 09:43:55AM -0600, Alex Williamson wrote:
> I was tempted to do this as well, but what about
> domain_remove_dev_info() where we also handle vm domains specially and
> don't do a symmetric detach in both directions?  I started down that
> path but quickly found the code to fragile to make those kinds of
> changes.  Thanks,

Okay, so domain_remove_dev_info() does:

                if (domain_type_is_vm(domain)) {
                        iommu_detach_dependent_devices(info->iommu, info->dev);
                        domain_detach_iommu(domain, info->iommu);
                }

... in a loop over all devices attached to the domain. The first
function (iommu_detach_dependent_devices) is not special to vm-domains.
In fact, domain_remove_one_dev_info calls it for all domain types.

And domain_detach_iommu only clears the bit in the iommu_bmp of the
domain, which is also not special and also called for all domain types
in domain_remove_one_dev_info.

So it looks like this special handling has no real purpose and is just
part of the whole mess. I am currently working on the conversion of the
Intel VT-d driver to use default-domains from the iommu-core. When this
is done we can get rid of that mess.



	Joerg

Re: [PATCH] iommu/vt-d: Fix VM domain ID leak

[Alex Williamson]

On Thu, 2015-07-16 at 19:03 +0200, Joerg Roedel wrote:
> On Thu, Jul 16, 2015 at 09:43:55AM -0600, Alex Williamson wrote:
> > I was tempted to do this as well, but what about
> > domain_remove_dev_info() where we also handle vm domains specially and
> > don't do a symmetric detach in both directions?  I started down that
> > path but quickly found the code to fragile to make those kinds of
> > changes.  Thanks,
> 
> Okay, so domain_remove_dev_info() does:
> 
>                 if (domain_type_is_vm(domain)) {
>                         iommu_detach_dependent_devices(info->iommu, info->dev);
>                         domain_detach_iommu(domain, info->iommu);
>                 }
> 
> ... in a loop over all devices attached to the domain. The first
> function (iommu_detach_dependent_devices) is not special to vm-domains.
> In fact, domain_remove_one_dev_info calls it for all domain types.
> 
> And domain_detach_iommu only clears the bit in the iommu_bmp of the
> domain, which is also not special and also called for all domain types
> in domain_remove_one_dev_info.
> 
> So it looks like this special handling has no real purpose and is just
> part of the whole mess. I am currently working on the conversion of the
> Intel VT-d driver to use default-domains from the iommu-core. When this
> is done we can get rid of that mess.

In the meantime, we're still leaking domain IDs, do you see any problem
with the proposed patch for 4.2 and stable?  Thanks,

Alex

Re: [PATCH] iommu/vt-d: Fix VM domain ID leak

[Joerg Roedel]

On Wed, Jul 22, 2015 at 03:01:04PM -0600, Alex Williamson wrote:
> In the meantime, we're still leaking domain IDs, do you see any problem
> with the proposed patch for 4.2 and stable?  Thanks,

Applied and pull-request sent.