[PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Robbie VanVossen]

If multiple devices are being passed through to the same domain and they
share a single SMMU, then they only require a single iommu_domain.

In arm_smmu_assign_dev, before a new iommu_domain is created, the
xen_domain->contexts is checked for any iommu_domains that are already
assigned to device that uses the same SMMU as the current device. If one
is found, attach the device to that iommu_domain. If a new one isn't
found, create a new iommu_domain just like before.

The arm_smmu_deassign_dev function assumes that there is a single
device per iommu_domain. This meant that when the first device was
deassigned, the iommu_domain was freed and when another device was
deassigned a crash occured in xen.

To fix this, a reference counter was added to the iommu_domain struct.
When an arm_smmu_xen_device references an iommu_domain, the
iommu_domains ref is incremented. When that reference is removed, the
iommu_domains ref is decremented. The iommu_domain will only be freed
when the ref is 0.

Signed-off-by: Robbie VanVossen <robert.vanvossen@dornerworks.com>

---
Changed since v1:
  * Fixed coding style for comments
  * Move increment/decrement outside of attach/detach functions
  * Expanded xen_domain->lock to protect more of the assign/deassign
    functions
  * Removed iommu_domain add/remove_device functions
---
 xen/drivers/passthrough/arm/smmu.c |   98 +++++++++++++++++++++++++++---------
 1 file changed, 74 insertions(+), 24 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index a7a7da9..1a3de00 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -223,6 +223,7 @@ struct iommu_domain
 	/* Runtime SMMU configuration for this iommu_domain */
 	struct arm_smmu_domain		*priv;
 
+	atomic_t ref;
 	/* Used to link iommu_domain contexts for a same domain.
 	 * There is at least one per-SMMU to used by the domain.
 	 * */
@@ -2569,7 +2570,9 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 {
 	struct iommu_domain *domain;
 	struct arm_smmu_xen_domain *xen_domain;
-	int ret;
+	struct arm_smmu_device *smmu;
+	int ret = 0;
+	int existing_ctxt_fnd = 0;
 
 	xen_domain = domain_hvm_iommu(d)->arch.priv;
 
@@ -2585,36 +2588,77 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
 			return ret;
 	}
 
+	spin_lock(&xen_domain->lock);
+
 	/*
-	 * TODO: Share the context bank (i.e iommu_domain) when the device is
-	 * under the same SMMU as another device assigned to this domain.
-	 * Would it useful for PCI
+	 * Check to see if a context bank (iommu_domain) already exists for
+	 * this xen domain under the same SMMU
 	 */
-	domain = xzalloc(struct iommu_domain);
-	if (!domain)
-		return -ENOMEM;
+	if (!list_empty(&xen_domain->contexts)) {
+		smmu = find_smmu_for_device(dev);
+		if (!smmu) {
+			dev_err(dev, "cannot find SMMU\n");
+			ret = -ENXIO;
+			goto out;
+		}
 
-	ret = arm_smmu_domain_init(domain);
-	if (ret)
-		goto err_dom_init;
+		/*
+		 * Loop through the &xen_domain->contexts to locate a context
+		 * assigned to this SMMU
+		 */
+		list_for_each_entry(domain, &xen_domain->contexts, list) {
+			if(domain->priv->smmu == smmu)
+			{
+				/*
+				 * We have found a context already associated with the
+				 * same xen domain and SMMU
+				 */
+				ret = arm_smmu_attach_dev(domain, dev);
+				if (ret) {
+					dev_err(dev,
+						"cannot attach device to already existing iommu_domain\n");
+					goto out;
+				}
+				atomic_inc(&domain->ref);
+
+				existing_ctxt_fnd = 1;
+				break;
+			}
+		}
+	}
 
-	domain->priv->cfg.domain = d;
+	if(!existing_ctxt_fnd){
 
-	ret = arm_smmu_attach_dev(domain, dev);
-	if (ret)
-		goto err_attach_dev;
+		domain = xzalloc(struct iommu_domain);
+		if (!domain) {
+			ret = -ENOMEM;
+			goto out;
+		}
 
-	spin_lock(&xen_domain->lock);
-	/* Chain the new context to the domain */
-	list_add(&domain->list, &xen_domain->contexts);
-	spin_unlock(&xen_domain->lock);
+		ret = arm_smmu_domain_init(domain);
+		if (ret)
+			goto err_dom_init;
 
-	return 0;
+		domain->priv->cfg.domain = d;
+
+		ret = arm_smmu_attach_dev(domain, dev);
+		if (ret)
+			goto err_attach_dev;
+		atomic_inc(&domain->ref);
+
+		/* Chain the new context to the domain */
+		list_add(&domain->list, &xen_domain->contexts);
+		
+	}
+
+	goto out;
 
 err_attach_dev:
 	arm_smmu_domain_destroy(domain);
 err_dom_init:
 	xfree(domain);
+out:
+	spin_unlock(&xen_domain->lock);
 
 	return ret;
 }
@@ -2631,14 +2675,20 @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
 		return -ESRCH;
 	}
 
+	spin_lock(&xen_domain->lock);
+
 	arm_smmu_detach_dev(domain, dev);
+	atomic_dec(&domain->ref);
 
-	spin_lock(&xen_domain->lock);
-	list_del(&domain->list);
-	spin_unlock(&xen_domain->lock);
+	if (domain->ref.counter == 0)
+	{
+		list_del(&domain->list);
 
-	arm_smmu_domain_destroy(domain);
-	xfree(domain);
+		arm_smmu_domain_destroy(domain);
+		xfree(domain);
+	}
+
+	spin_unlock(&xen_domain->lock);
 
 	return 0;
 }
-- 
1.7.9.5

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Julien Grall]

Hello Robert,

On 23/03/2015 13:46, Robbie VanVossen wrote:
> If multiple devices are being passed through to the same domain and they
> share a single SMMU, then they only require a single iommu_domain.
>
> In arm_smmu_assign_dev, before a new iommu_domain is created, the
> xen_domain->contexts is checked for any iommu_domains that are already
> assigned to device that uses the same SMMU as the current device. If one
> is found, attach the device to that iommu_domain. If a new one isn't
> found, create a new iommu_domain just like before.
>
> The arm_smmu_deassign_dev function assumes that there is a single
> device per iommu_domain. This meant that when the first device was
> deassigned, the iommu_domain was freed and when another device was
> deassigned a crash occured in xen.
>
> To fix this, a reference counter was added to the iommu_domain struct.
> When an arm_smmu_xen_device references an iommu_domain, the
> iommu_domains ref is incremented. When that reference is removed, the
> iommu_domains ref is decremented. The iommu_domain will only be freed
> when the ref is 0.
>
> Signed-off-by: Robbie VanVossen <robert.vanvossen@dornerworks.com>
>
> ---
> Changed since v1:
>    * Fixed coding style for comments
>    * Move increment/decrement outside of attach/detach functions
>    * Expanded xen_domain->lock to protect more of the assign/deassign
>      functions
>    * Removed iommu_domain add/remove_device functions
> ---
>   xen/drivers/passthrough/arm/smmu.c |   98 +++++++++++++++++++++++++++---------
>   1 file changed, 74 insertions(+), 24 deletions(-)
>
> diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
> index a7a7da9..1a3de00 100644
> --- a/xen/drivers/passthrough/arm/smmu.c
> +++ b/xen/drivers/passthrough/arm/smmu.c
> @@ -223,6 +223,7 @@ struct iommu_domain
>   	/* Runtime SMMU configuration for this iommu_domain */
>   	struct arm_smmu_domain		*priv;
>
> +	atomic_t ref;
>   	/* Used to link iommu_domain contexts for a same domain.
>   	 * There is at least one per-SMMU to used by the domain.
>   	 * */
> @@ -2569,7 +2570,9 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>   {
>   	struct iommu_domain *domain;
>   	struct arm_smmu_xen_domain *xen_domain;
> -	int ret;
> +	struct arm_smmu_device *smmu;
> +	int ret = 0;
> +	int existing_ctxt_fnd = 0;

You can use a bool_t here.

>
>   	xen_domain = domain_hvm_iommu(d)->arch.priv;
>
> @@ -2585,36 +2588,77 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>   			return ret;
>   	}
>
> +	spin_lock(&xen_domain->lock);
> +
>   	/*
> -	 * TODO: Share the context bank (i.e iommu_domain) when the device is
> -	 * under the same SMMU as another device assigned to this domain.
> -	 * Would it useful for PCI
> +	 * Check to see if a context bank (iommu_domain) already exists for
> +	 * this xen domain under the same SMMU
>   	 */
> -	domain = xzalloc(struct iommu_domain);
> -	if (!domain)
> -		return -ENOMEM;
> +	if (!list_empty(&xen_domain->contexts)) {
> +		smmu = find_smmu_for_device(dev);
> +		if (!smmu) {
> +			dev_err(dev, "cannot find SMMU\n");
> +			ret = -ENXIO;
> +			goto out;
> +		}
>
> -	ret = arm_smmu_domain_init(domain);
> -	if (ret)
> -		goto err_dom_init;
> +		/*
> +		 * Loop through the &xen_domain->contexts to locate a context
> +		 * assigned to this SMMU
> +		 */
> +		list_for_each_entry(domain, &xen_domain->contexts, list) {
> +			if(domain->priv->smmu == smmu)
> +			{

Coding style. This should be
if (domain->priv->smmu == smmu) {

> +				/*
> +				 * We have found a context already associated with the
> +				 * same xen domain and SMMU
> +				 */
> +				ret = arm_smmu_attach_dev(domain, dev);
> +				if (ret) {
> +					dev_err(dev,
> +						"cannot attach device to already existing iommu_domain\n");

We don't print an error message for new domain, I don't think this one 
is useful.

> +					goto out;
> +				}
> +				atomic_inc(&domain->ref);

Introducing an helper to get the iommu_domain would simply the workflow.

That would give smth like:

domain = arm_smmu_get_domain(d, dev);
if (!domain)
{
     allocate domain
     Add to list dev
}

arm_smmu_attach_dev(domain, dev);
atomic_inc(&domain->ref);

out:
    if (failed && atomic == 0)
      destroy iommu_domain.

The destroy iommu_domain could be shared with the one in deassign_dev.


> +
> +				existing_ctxt_fnd = 1;
> +				break;
> +			}
> +		}
> +	}
>
> -	domain->priv->cfg.domain = d;
> +	if(!existing_ctxt_fnd){
>
> -	ret = arm_smmu_attach_dev(domain, dev);
> -	if (ret)
> -		goto err_attach_dev;
> +		domain = xzalloc(struct iommu_domain);
> +		if (!domain) {
> +			ret = -ENOMEM;
> +			goto out;
> +		}
>
> -	spin_lock(&xen_domain->lock);
> -	/* Chain the new context to the domain */
> -	list_add(&domain->list, &xen_domain->contexts);
> -	spin_unlock(&xen_domain->lock);
> +		ret = arm_smmu_domain_init(domain);
> +		if (ret)
> +			goto err_dom_init;
>
> -	return 0;
> +		domain->priv->cfg.domain = d;
> +
> +		ret = arm_smmu_attach_dev(domain, dev);
> +		if (ret)
> +			goto err_attach_dev;
> +		atomic_inc(&domain->ref);
> +
> +		/* Chain the new context to the domain */
> +		list_add(&domain->list, &xen_domain->contexts);
> +		
> +	}
> +
> +	goto out;
>
>   err_attach_dev:
>   	arm_smmu_domain_destroy(domain);
>   err_dom_init:
>   	xfree(domain);
> +out:
> +	spin_unlock(&xen_domain->lock);
>
>   	return ret;
>   }
> @@ -2631,14 +2675,20 @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
>   		return -ESRCH;
>   	}
>
> +	spin_lock(&xen_domain->lock);
> +
>   	arm_smmu_detach_dev(domain, dev);
> +	atomic_dec(&domain->ref);
>
> -	spin_lock(&xen_domain->lock);
> -	list_del(&domain->list);
> -	spin_unlock(&xen_domain->lock);
> +	if (domain->ref.counter == 0)
> +	{

Coding style:

if (...) {

> +		list_del(&domain->list);
>
> -	arm_smmu_domain_destroy(domain);
> -	xfree(domain);
> +		arm_smmu_domain_destroy(domain);
> +		xfree(domain);
> +	}
> +
> +	spin_unlock(&xen_domain->lock);
>
>   	return 0;
>   }
>

Regards,


-- 
Julien Grall

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Robert VanVossen]

Hello Julien,

On 3/23/2015 5:53 PM, Julien Grall wrote:
> Hello Robert,
> 
> On 23/03/2015 13:46, Robbie VanVossen wrote:
>> @@ -2569,7 +2570,9 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>>   {
>>   	struct iommu_domain *domain;
>>   	struct arm_smmu_xen_domain *xen_domain;
>> -	int ret;
>> +	struct arm_smmu_device *smmu;
>> +	int ret = 0;
>> +	int existing_ctxt_fnd = 0;
> 
> You can use a bool_t here.
> 
Ok.

>>
>>   	xen_domain = domain_hvm_iommu(d)->arch.priv;
>>
>> @@ -2585,36 +2588,77 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>>   			return ret;
>>   	}
>>
>> +	spin_lock(&xen_domain->lock);
>> +
>>   	/*
>> -	 * TODO: Share the context bank (i.e iommu_domain) when the device is
>> -	 * under the same SMMU as another device assigned to this domain.
>> -	 * Would it useful for PCI
>> +	 * Check to see if a context bank (iommu_domain) already exists for
>> +	 * this xen domain under the same SMMU
>>   	 */
>> -	domain = xzalloc(struct iommu_domain);
>> -	if (!domain)
>> -		return -ENOMEM;
>> +	if (!list_empty(&xen_domain->contexts)) {
>> +		smmu = find_smmu_for_device(dev);
>> +		if (!smmu) {
>> +			dev_err(dev, "cannot find SMMU\n");
>> +			ret = -ENXIO;
>> +			goto out;
>> +		}
>>
>> -	ret = arm_smmu_domain_init(domain);
>> -	if (ret)
>> -		goto err_dom_init;
>> +		/*
>> +		 * Loop through the &xen_domain->contexts to locate a context
>> +		 * assigned to this SMMU
>> +		 */
>> +		list_for_each_entry(domain, &xen_domain->contexts, list) {
>> +			if(domain->priv->smmu == smmu)
>> +			{
> 
> Coding style. This should be
> if (domain->priv->smmu == smmu) {
> 
Will fix.

>> +				/*
>> +				 * We have found a context already associated with the
>> +				 * same xen domain and SMMU
>> +				 */
>> +				ret = arm_smmu_attach_dev(domain, dev);
>> +				if (ret) {
>> +					dev_err(dev,
>> +						"cannot attach device to already existing iommu_domain\n");
> 
> We don't print an error message for new domain, I don't think this one 
> is useful.
> 
Will remove dev_err.

>> +					goto out;
>> +				}
>> +				atomic_inc(&domain->ref);
> 
> Introducing an helper to get the iommu_domain would simply the workflow.
> 
> That would give smth like:
> 
> domain = arm_smmu_get_domain(d, dev);
> if (!domain)
> {
>      allocate domain
>      Add to list dev
> }
> 
> arm_smmu_attach_dev(domain, dev);
> atomic_inc(&domain->ref);
> 
> out:
>     if (failed && atomic == 0)
>       destroy iommu_domain.
> 
> The destroy iommu_domain could be shared with the one in deassign_dev.
> 

But don't we need to handle the failure of attach_dev differently based on which
fails? If the attach_dev fails on a new iommu_domain, then the iommu_domain
should be destroyed. If it fails on a preexisting iommu_domain that is already
attached to another device, then you don't want to destroy the iommu_domain.

>> +
>> +				existing_ctxt_fnd = 1;
>> +				break;
>> +			}
>> +		}
>> +	}
>>
>> -	domain->priv->cfg.domain = d;
>> +	if(!existing_ctxt_fnd){
>>
>> -	ret = arm_smmu_attach_dev(domain, dev);
>> -	if (ret)
>> -		goto err_attach_dev;
>> +		domain = xzalloc(struct iommu_domain);
>> +		if (!domain) {
>> +			ret = -ENOMEM;
>> +			goto out;
>> +		}
>>
>> -	spin_lock(&xen_domain->lock);
>> -	/* Chain the new context to the domain */
>> -	list_add(&domain->list, &xen_domain->contexts);
>> -	spin_unlock(&xen_domain->lock);
>> +		ret = arm_smmu_domain_init(domain);
>> +		if (ret)
>> +			goto err_dom_init;
>>
>> -	return 0;
>> +		domain->priv->cfg.domain = d;
>> +
>> +		ret = arm_smmu_attach_dev(domain, dev);
>> +		if (ret)
>> +			goto err_attach_dev;
>> +		atomic_inc(&domain->ref);
>> +
>> +		/* Chain the new context to the domain */
>> +		list_add(&domain->list, &xen_domain->contexts);
>> +		
>> +	}
>> +
>> +	goto out;
>>
>>   err_attach_dev:
>>   	arm_smmu_domain_destroy(domain);
>>   err_dom_init:
>>   	xfree(domain);
>> +out:
>> +	spin_unlock(&xen_domain->lock);
>>
>>   	return ret;
>>   }
>> @@ -2631,14 +2675,20 @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
>>   		return -ESRCH;
>>   	}
>>
>> +	spin_lock(&xen_domain->lock);
>> +
>>   	arm_smmu_detach_dev(domain, dev);
>> +	atomic_dec(&domain->ref);
>>
>> -	spin_lock(&xen_domain->lock);
>> -	list_del(&domain->list);
>> -	spin_unlock(&xen_domain->lock);
>> +	if (domain->ref.counter == 0)
>> +	{
> 
> Coding style:
> 
> if (...) {
> 
Will Fix

Thanks,
Robbie VanVossen

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Julien Grall]

On 24/03/2015 15:07, Robert VanVossen wrote:
> Hello Julien,

Hello Robert,


> On 3/23/2015 5:53 PM, Julien Grall wrote:
> But don't we need to handle the failure of attach_dev differently based on which
> fails? If the attach_dev fails on a new iommu_domain, then the iommu_domain
> should be destroyed. If it fails on a preexisting iommu_domain that is already
> attached to another device, then you don't want to destroy the iommu_domain.

It should be covered by

if (failed && atomic == 0)
   destroy iommu_domain.

If you prefer to keep the 2 possible case separate, I would still like 
to have a separate helper to get the domain.

Regards,

-- 
Julien Grall

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Manish Jaggi]

On Monday 23 March 2015 07:16 PM, Robbie VanVossen wrote:
> If multiple devices are being passed through to the same domain and they
> share a single SMMU, then they only require a single iommu_domain.
>
> In arm_smmu_assign_dev, before a new iommu_domain is created, the
> xen_domain->contexts is checked for any iommu_domains that are already
> assigned to device that uses the same SMMU as the current device. If one
> is found, attach the device to that iommu_domain. If a new one isn't
> found, create a new iommu_domain just like before.
>
> The arm_smmu_deassign_dev function assumes that there is a single
> device per iommu_domain. This meant that when the first device was
> deassigned, the iommu_domain was freed and when another device was
> deassigned a crash occured in xen.
>
> To fix this, a reference counter was added to the iommu_domain struct.
> When an arm_smmu_xen_device references an iommu_domain, the
> iommu_domains ref is incremented. When that reference is removed, the
> iommu_domains ref is decremented. The iommu_domain will only be freed
> when the ref is 0.
>
> Signed-off-by: Robbie VanVossen <robert.vanvossen@dornerworks.com>
Hi,
Are you adding a PCI passthrough support to Xen ?. I am in process of 
sending smmu driver patches based on juliens latest code.

> ---
> Changed since v1:
>    * Fixed coding style for comments
>    * Move increment/decrement outside of attach/detach functions
>    * Expanded xen_domain->lock to protect more of the assign/deassign
>      functions
>    * Removed iommu_domain add/remove_device functions
> ---
>   xen/drivers/passthrough/arm/smmu.c |   98 +++++++++++++++++++++++++++---------
>   1 file changed, 74 insertions(+), 24 deletions(-)
>
> diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
> index a7a7da9..1a3de00 100644
> --- a/xen/drivers/passthrough/arm/smmu.c
> +++ b/xen/drivers/passthrough/arm/smmu.c
> @@ -223,6 +223,7 @@ struct iommu_domain
>   	/* Runtime SMMU configuration for this iommu_domain */
>   	struct arm_smmu_domain		*priv;
>   
> +	atomic_t ref;
>   	/* Used to link iommu_domain contexts for a same domain.
>   	 * There is at least one per-SMMU to used by the domain.
>   	 * */
> @@ -2569,7 +2570,9 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>   {
>   	struct iommu_domain *domain;
>   	struct arm_smmu_xen_domain *xen_domain;
> -	int ret;
> +	struct arm_smmu_device *smmu;
> +	int ret = 0;
> +	int existing_ctxt_fnd = 0;
>   
>   	xen_domain = domain_hvm_iommu(d)->arch.priv;
>   
> @@ -2585,36 +2588,77 @@ static int arm_smmu_assign_dev(struct domain *d, u8 devfn,
>   			return ret;
>   	}
>   
> +	spin_lock(&xen_domain->lock);
> +
>   	/*
> -	 * TODO: Share the context bank (i.e iommu_domain) when the device is
> -	 * under the same SMMU as another device assigned to this domain.
> -	 * Would it useful for PCI
> +	 * Check to see if a context bank (iommu_domain) already exists for
> +	 * this xen domain under the same SMMU
>   	 */
> -	domain = xzalloc(struct iommu_domain);
> -	if (!domain)
> -		return -ENOMEM;
> +	if (!list_empty(&xen_domain->contexts)) {
> +		smmu = find_smmu_for_device(dev);
> +		if (!smmu) {
> +			dev_err(dev, "cannot find SMMU\n");
> +			ret = -ENXIO;
> +			goto out;
> +		}
>   
> -	ret = arm_smmu_domain_init(domain);
> -	if (ret)
> -		goto err_dom_init;
> +		/*
> +		 * Loop through the &xen_domain->contexts to locate a context
> +		 * assigned to this SMMU
> +		 */
> +		list_for_each_entry(domain, &xen_domain->contexts, list) {
> +			if(domain->priv->smmu == smmu)
> +			{
> +				/*
> +				 * We have found a context already associated with the
> +				 * same xen domain and SMMU
> +				 */
> +				ret = arm_smmu_attach_dev(domain, dev);
> +				if (ret) {
> +					dev_err(dev,
> +						"cannot attach device to already existing iommu_domain\n");
> +					goto out;
> +				}
> +				atomic_inc(&domain->ref);
> +
> +				existing_ctxt_fnd = 1;
> +				break;
> +			}
> +		}
> +	}
>   
> -	domain->priv->cfg.domain = d;
> +	if(!existing_ctxt_fnd){
>   
> -	ret = arm_smmu_attach_dev(domain, dev);
> -	if (ret)
> -		goto err_attach_dev;
> +		domain = xzalloc(struct iommu_domain);
> +		if (!domain) {
> +			ret = -ENOMEM;
> +			goto out;
> +		}
>   
> -	spin_lock(&xen_domain->lock);
> -	/* Chain the new context to the domain */
> -	list_add(&domain->list, &xen_domain->contexts);
> -	spin_unlock(&xen_domain->lock);
> +		ret = arm_smmu_domain_init(domain);
> +		if (ret)
> +			goto err_dom_init;
>   
> -	return 0;
> +		domain->priv->cfg.domain = d;
> +
> +		ret = arm_smmu_attach_dev(domain, dev);
> +		if (ret)
> +			goto err_attach_dev;
> +		atomic_inc(&domain->ref);
> +
> +		/* Chain the new context to the domain */
> +		list_add(&domain->list, &xen_domain->contexts);
> +		
> +	}
> +
> +	goto out;
>   
>   err_attach_dev:
>   	arm_smmu_domain_destroy(domain);
>   err_dom_init:
>   	xfree(domain);
> +out:
> +	spin_unlock(&xen_domain->lock);
>   
>   	return ret;
>   }
> @@ -2631,14 +2675,20 @@ static int arm_smmu_deassign_dev(struct domain *d, struct device *dev)
>   		return -ESRCH;
>   	}
>   
> +	spin_lock(&xen_domain->lock);
> +
>   	arm_smmu_detach_dev(domain, dev);
> +	atomic_dec(&domain->ref);
>   
> -	spin_lock(&xen_domain->lock);
> -	list_del(&domain->list);
> -	spin_unlock(&xen_domain->lock);
> +	if (domain->ref.counter == 0)
> +	{
> +		list_del(&domain->list);
>   
> -	arm_smmu_domain_destroy(domain);
> -	xfree(domain);
> +		arm_smmu_domain_destroy(domain);
> +		xfree(domain);
> +	}
> +
> +	spin_unlock(&xen_domain->lock);
>   
>   	return 0;
>   }

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Robert VanVossen]

On 3/24/2015 7:07 AM, Manish Jaggi wrote:
> 
> On Monday 23 March 2015 07:16 PM, Robbie VanVossen wrote:
>> If multiple devices are being passed through to the same domain and they
>> share a single SMMU, then they only require a single iommu_domain.
>>
>> In arm_smmu_assign_dev, before a new iommu_domain is created, the
>> xen_domain->contexts is checked for any iommu_domains that are already
>> assigned to device that uses the same SMMU as the current device. If one
>> is found, attach the device to that iommu_domain. If a new one isn't
>> found, create a new iommu_domain just like before.
>>
>> The arm_smmu_deassign_dev function assumes that there is a single
>> device per iommu_domain. This meant that when the first device was
>> deassigned, the iommu_domain was freed and when another device was
>> deassigned a crash occured in xen.
>>
>> To fix this, a reference counter was added to the iommu_domain struct.
>> When an arm_smmu_xen_device references an iommu_domain, the
>> iommu_domains ref is incremented. When that reference is removed, the
>> iommu_domains ref is decremented. The iommu_domain will only be freed
>> when the ref is 0.
>>
>> Signed-off-by: Robbie VanVossen <robert.vanvossen@dornerworks.com>
> Hi,
> Are you adding a PCI passthrough support to Xen ?. I am in process of 
> sending smmu driver patches based on juliens latest code.
>

Nope, I am just working on what this patch describes

Thanks,
Robbie VanVossen
DornerWorks

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Manish Jaggi]

On Tuesday 24 March 2015 07:34 PM, Robert VanVossen wrote:
>
> On 3/24/2015 7:07 AM, Manish Jaggi wrote:
>> On Monday 23 March 2015 07:16 PM, Robbie VanVossen wrote:
>>> If multiple devices are being passed through to the same domain and they
>>> share a single SMMU, then they only require a single iommu_domain.
>>>
>>> In arm_smmu_assign_dev, before a new iommu_domain is created, the
>>> xen_domain->contexts is checked for any iommu_domains that are already
>>> assigned to device that uses the same SMMU as the current device. If one
>>> is found, attach the device to that iommu_domain. If a new one isn't
>>> found, create a new iommu_domain just like before.
>>>
>>> The arm_smmu_deassign_dev function assumes that there is a single
>>> device per iommu_domain. This meant that when the first device was
>>> deassigned, the iommu_domain was freed and when another device was
>>> deassigned a crash occured in xen.
>>>
>>> To fix this, a reference counter was added to the iommu_domain struct.
>>> When an arm_smmu_xen_device references an iommu_domain, the
>>> iommu_domains ref is incremented. When that reference is removed, the
>>> iommu_domains ref is decremented. The iommu_domain will only be freed
>>> when the ref is 0.
>>>
>>> Signed-off-by: Robbie VanVossen <robert.vanvossen@dornerworks.com>
>> Hi,
>> Are you adding a PCI passthrough support to Xen ?. I am in process of
>> sending smmu driver patches based on juliens latest code.
>>
> Nope, I am just working on what this patch describes
Your patch is doing similar thing what am working on. As per the Xen 4.6 
release I took smmu for pci-passthorugh. I didnt send smmu patches so 
far as there are other dependency patches I need to send first like 
pci_host_bridge register hypercall patch.

Julien/Ian could you please hold the merge of this patch.
> Thanks,
> Robbie VanVossen
> DornerWorks
>

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Stefano Stabellini]

On Wed, 25 Mar 2015, Manish Jaggi wrote:
> On Tuesday 24 March 2015 07:34 PM, Robert VanVossen wrote:
> > 
> > On 3/24/2015 7:07 AM, Manish Jaggi wrote:
> > > On Monday 23 March 2015 07:16 PM, Robbie VanVossen wrote:
> > > > If multiple devices are being passed through to the same domain and they
> > > > share a single SMMU, then they only require a single iommu_domain.
> > > > 
> > > > In arm_smmu_assign_dev, before a new iommu_domain is created, the
> > > > xen_domain->contexts is checked for any iommu_domains that are already
> > > > assigned to device that uses the same SMMU as the current device. If one
> > > > is found, attach the device to that iommu_domain. If a new one isn't
> > > > found, create a new iommu_domain just like before.
> > > > 
> > > > The arm_smmu_deassign_dev function assumes that there is a single
> > > > device per iommu_domain. This meant that when the first device was
> > > > deassigned, the iommu_domain was freed and when another device was
> > > > deassigned a crash occured in xen.
> > > > 
> > > > To fix this, a reference counter was added to the iommu_domain struct.
> > > > When an arm_smmu_xen_device references an iommu_domain, the
> > > > iommu_domains ref is incremented. When that reference is removed, the
> > > > iommu_domains ref is decremented. The iommu_domain will only be freed
> > > > when the ref is 0.
> > > > 
> > > > Signed-off-by: Robbie VanVossen <robert.vanvossen@dornerworks.com>
> > > Hi,
> > > Are you adding a PCI passthrough support to Xen ?. I am in process of
> > > sending smmu driver patches based on juliens latest code.
> > > 
> > Nope, I am just working on what this patch describes
> Your patch is doing similar thing what am working on. As per the Xen 4.6
> release I took smmu for pci-passthorugh. I didnt send smmu patches so far as
> there are other dependency patches I need to send first like pci_host_bridge
> register hypercall patch.
> 
> Julien/Ian could you please hold the merge of this patch.

Sorry Manish, but the Xen community works on a first come first served
basis:  if/when this patch is considered in good condition and ready to
be merged, is likely to be merged.
Unless you have a specific concern about the code of course.

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Manish Jaggi]

On Wednesday 25 March 2015 11:05 PM, Stefano Stabellini wrote:
> On Wed, 25 Mar 2015, Manish Jaggi wrote:
>> On Tuesday 24 March 2015 07:34 PM, Robert VanVossen wrote:
>>> On 3/24/2015 7:07 AM, Manish Jaggi wrote:
>>>> On Monday 23 March 2015 07:16 PM, Robbie VanVossen wrote:
>>>>> If multiple devices are being passed through to the same domain and they
>>>>> share a single SMMU, then they only require a single iommu_domain.
>>>>>
>>>>> In arm_smmu_assign_dev, before a new iommu_domain is created, the
>>>>> xen_domain->contexts is checked for any iommu_domains that are already
>>>>> assigned to device that uses the same SMMU as the current device. If one
>>>>> is found, attach the device to that iommu_domain. If a new one isn't
>>>>> found, create a new iommu_domain just like before.
>>>>>
>>>>> The arm_smmu_deassign_dev function assumes that there is a single
>>>>> device per iommu_domain. This meant that when the first device was
>>>>> deassigned, the iommu_domain was freed and when another device was
>>>>> deassigned a crash occured in xen.
>>>>>
>>>>> To fix this, a reference counter was added to the iommu_domain struct.
>>>>> When an arm_smmu_xen_device references an iommu_domain, the
>>>>> iommu_domains ref is incremented. When that reference is removed, the
>>>>> iommu_domains ref is decremented. The iommu_domain will only be freed
>>>>> when the ref is 0.
>>>>>
>>>>> Signed-off-by: Robbie VanVossen <robert.vanvossen@dornerworks.com>
>>>> Hi,
>>>> Are you adding a PCI passthrough support to Xen ?. I am in process of
>>>> sending smmu driver patches based on juliens latest code.
>>>>
>>> Nope, I am just working on what this patch describes
>> Your patch is doing similar thing what am working on. As per the Xen 4.6
>> release I took smmu for pci-passthorugh. I didnt send smmu patches so far as
>> there are other dependency patches I need to send first like pci_host_bridge
>> register hypercall patch.
>>
>> Julien/Ian could you please hold the merge of this patch.
> Sorry Manish, but the Xen community works on a first come first served
> basis:  if/when this patch is considered in good condition and ready to
> be merged, is likely to be merged.
> Unless you have a specific concern about the code of course.
Ok. I will incorporate robert's patch in my patch set.
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel

Re: [PATCH v2] xen/passthrough: Support a single iommu_domain per xen domain per SMMU

[Julien Grall]

Hello,

On 30/03/15 12:04, Manish Jaggi wrote:
>>> Julien/Ian could you please hold the merge of this patch.
>> Sorry Manish, but the Xen community works on a first come first served
>> basis:  if/when this patch is considered in good condition and ready to
>> be merged, is likely to be merged.
>> Unless you have a specific concern about the code of course.
> Ok. I will incorporate robert's patch in my patch set.

Why we should wait you to sent your PCI SMMU changes? It will likely
take another couple of months before your series will be merged.

Robert's patch is a standalone patch which is useful for non-PCI case
(for instance 2 networks cards under the same SMMU).

He sent the v4 last week and the patch is ready to go in Xen tree.

Regards,

-- 
Julien Grall