[RFC 0/2] Cap SMMU s2 input-size based on p2m tables

[RFC 0/2] Cap SMMU s2 input-size based on p2m tables

[Edgar E. Iglesias]

From: "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>

Hi,

This is a first try at fixing the issue I'm seeing on ZynqMP with
missmatched setup of the SMMU and the shared page-tables with p2m.

I've only handled the case of having an SMMU that supports larger
s2 input-sizes than what p2m wants to use.

To support the case were the system has SMMUs that can only do smaller
input-sizes than the CPU, I think we will need to walk the device-tree
when creating domains and somehow cap the p2m IPA size to the largest
supported S2 size across the SMMUs used by that particular domain.
The max IPA size may need to be domain specific.
I did not implement this but I did make the max size domain specific.
Julien, you mentioned you might have access to such a system?

Does this look reasonable or should we be fixing this some other way?
I could make the patch simpler I guess by having a global (not
domain specific) max value set once aswell.

Comments?

Best regards,
Edgar

Edgar E. Iglesias (2):
  xen/arm: Save and expose the p2m IPA address size
  xen/iommu: smmu: Use the p2m IPA size as S2 input-size

 xen/arch/arm/p2m.c                 |  7 ++++++-
 xen/drivers/passthrough/arm/smmu.c | 17 +++++++++++++++--
 xen/include/asm-arm/p2m.h          |  3 +++
 3 files changed, 24 insertions(+), 3 deletions(-)

-- 
1.9.1

[RFC 1/2] xen/arm: Save and expose the p2m IPA address size

[Edgar E. Iglesias]

From: "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>

Save and expose the p2m IPA address size to arch specific code.
This is primarily needed to setup SMMUs as these share p2m
tables with guest CPUs.

Signed-off-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
---
 xen/arch/arm/p2m.c        | 7 ++++++-
 xen/include/asm-arm/p2m.h | 3 +++
 2 files changed, 9 insertions(+), 1 deletion(-)

diff --git a/xen/arch/arm/p2m.c b/xen/arch/arm/p2m.c
index 8dfee24..58df568 100644
--- a/xen/arch/arm/p2m.c
+++ b/xen/arch/arm/p2m.c
@@ -17,12 +17,15 @@
 #ifdef CONFIG_ARM_64
 static unsigned int __read_mostly p2m_root_order;
 static unsigned int __read_mostly p2m_root_level;
+static unsigned int __read_mostly p2m_ipa_bits;
 #define P2M_ROOT_ORDER    p2m_root_order
 #define P2M_ROOT_LEVEL p2m_root_level
+#define P2M_IPA_BITS p2m_ipa_bits
 #else
 /* First level P2M is alway 2 consecutive pages */
 #define P2M_ROOT_LEVEL 1
 #define P2M_ROOT_ORDER    1
+#define P2M_IPA_BITS 40
 #endif
 
 #define P2M_ROOT_PAGES    (1<<P2M_ROOT_ORDER)
@@ -1310,6 +1313,7 @@ int p2m_init(struct domain *d)
 
     p2m->root = NULL;
 
+    p2m->ipa_bits = P2M_IPA_BITS;
     p2m->max_mapped_gfn = 0;
     p2m->lowest_mapped_gfn = ULONG_MAX;
 
@@ -1557,9 +1561,10 @@ void __init setup_virt_paging(void)
 
     p2m_root_order = pa_range_info[pa_range].root_order;
     p2m_root_level = 2 - pa_range_info[pa_range].sl0;
+    p2m_ipa_bits = 64 - pa_range_info[pa_range].t0sz;
 
     printk("P2M: %d-bit IPA with %d-bit PA\n",
-           64 - pa_range_info[pa_range].t0sz,
+           p2m_ipa_bits,
            pa_range_info[pa_range].pabits);
 #endif
     printk("P2M: %d levels with order-%d root, VTCR 0x%lx\n",
diff --git a/xen/include/asm-arm/p2m.h b/xen/include/asm-arm/p2m.h
index 341df55..813ddd0 100644
--- a/xen/include/asm-arm/p2m.h
+++ b/xen/include/asm-arm/p2m.h
@@ -28,6 +28,9 @@ struct p2m_domain {
     /* Current VMID in use */
     uint8_t vmid;
 
+    /* Nr of bits in an IPA.  */
+    unsigned int ipa_bits;
+
     /* Highest guest frame that's ever been mapped in the p2m
      * Only takes into account ram and foreign mapping
      */
-- 
1.9.1

[RFC 2/2] xen/iommu: smmu: Use the p2m IPA size as S2 input-size

[Edgar E. Iglesias]

From: "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>

Because we share the p2m tables between SMMU and CPUs,
we need to make sure the SMMU is configured to use the
same S2 input-size as the CPU.

Signed-off-by: Edgar E. Iglesias <edgar.iglesias@xilinx.com>
---
 xen/drivers/passthrough/arm/smmu.c | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)

diff --git a/xen/drivers/passthrough/arm/smmu.c b/xen/drivers/passthrough/arm/smmu.c
index 8a9b58b..95e4dce 100644
--- a/xen/drivers/passthrough/arm/smmu.c
+++ b/xen/drivers/passthrough/arm/smmu.c
@@ -1038,6 +1038,7 @@ static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain)
 	struct arm_smmu_device *smmu = smmu_domain->smmu;
 	void __iomem *cb_base, *gr0_base, *gr1_base;
 	paddr_t p2maddr;
+	unsigned p2m_ipa_bits;
 
 	gr0_base = ARM_SMMU_GR0(smmu);
 	gr1_base = ARM_SMMU_GR1(smmu);
@@ -1123,6 +1124,14 @@ static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain)
 	/* TTBR0 */
 	/* Xen: The page table is shared with the P2M code */
 	ASSERT(smmu_domain->cfg.domain != NULL);
+
+	/* Xen: When building the domain, we need to make sure that the p2m
+	 * IPA size is not larger than what SMMUs can handle. It's to late to
+	 * do anything about it at this stage, so we simply ASSERT on it.
+	 */
+	p2m_ipa_bits = smmu_domain->cfg.domain->arch.p2m.ipa_bits;
+	ASSERT(smmu->s2_input_size >= p2m_ipa_bits);
+
 	p2maddr = page_to_maddr(smmu_domain->cfg.domain->arch.p2m.root);
 
 	dev_notice(smmu->dev, "d%u: p2maddr 0x%"PRIpaddr"\n",
@@ -1146,7 +1155,10 @@ static void arm_smmu_init_context_bank(struct arm_smmu_domain *smmu_domain)
 			reg = TTBCR_TG0_64K;
 
 		if (!stage1) {
-			reg |= (64 - smmu->s2_input_size) << TTBCR_T0SZ_SHIFT;
+			/* Xen: We reuse the p2m tables, the input size needs
+			 * to match the p2m IPA size.
+			 */
+			reg |= (64 - p2m_ipa_bits) << TTBCR_T0SZ_SHIFT;
 
 			switch (smmu->s2_output_size) {
 			case 32:
@@ -1795,10 +1807,11 @@ static int arm_smmu_handle_mapping(struct arm_smmu_domain *smmu_domain,
 	struct arm_smmu_cfg *cfg = &smmu_domain->cfg;
 	pgd_t *pgd = cfg->pgd;
 	unsigned long flags;
+	unsigned int p2m_ipa_bits = smmu_domain->cfg.domain->arch.p2m.ipa_bits;
 
 	if (cfg->cbar == CBAR_TYPE_S2_TRANS) {
 		stage = 2;
-		input_mask = (1ULL << smmu->s2_input_size) - 1;
+		input_mask = (1ULL << p2m_ipa_bits) - 1;
 		output_mask = (1ULL << smmu->s2_output_size) - 1;
 	} else {
 		stage = 1;
-- 
1.9.1

Re: [RFC 0/2] Cap SMMU s2 input-size based on p2m tables

[Julien Grall]

On 27/04/15 10:24, Edgar E. Iglesias wrote:
> From: "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>
> 
> Hi,

Hi Edgar,

> This is a first try at fixing the issue I'm seeing on ZynqMP with
> missmatched setup of the SMMU and the shared page-tables with p2m.
> 
> I've only handled the case of having an SMMU that supports larger
> s2 input-sizes than what p2m wants to use.
> 
> To support the case were the system has SMMUs that can only do smaller
> input-sizes than the CPU, I think we will need to walk the device-tree
> when creating domains and somehow cap the p2m IPA size to the largest
> supported S2 size across the SMMUs used by that particular domain.
> The max IPA size may need to be domain specific.
> I did not implement this but I did make the max size domain specific.
> Julien, you mentioned you might have access to such a system?

Right, I got an AMD Seattle where the IPA is limited by the SMMU. From
Xen output:
   - SMMU: 40-bit IPA -> 40-bit PA
   - P2M: 44-bit IPA with 44-bit PA

Suravee, can you confirm that this is actually the case?

> 
> Does this look reasonable or should we be fixing this some other way?
> I could make the patch simpler I guess by having a global (not
> domain specific) max value set once aswell.

For your use case a global seems better (you may need to move the
iommu_setup call after setup_virt_paging).

Although, if the limitation of the SMMU on Seattle is confirmed, that
would restrict domain memory to 1TB maximum (rather than 16TB) on this
platform.

Regards,

-- 
Julien Grall

Re: [RFC 0/2] Cap SMMU s2 input-size based on p2m tables

[Edgar E. Iglesias]

On Tue, Apr 28, 2015 at 05:50:43PM +0100, Julien Grall wrote:
> On 27/04/15 10:24, Edgar E. Iglesias wrote:
> > From: "Edgar E. Iglesias" <edgar.iglesias@xilinx.com>
> > 
> > Hi,
> 
> Hi Edgar,
> 
> > This is a first try at fixing the issue I'm seeing on ZynqMP with
> > missmatched setup of the SMMU and the shared page-tables with p2m.
> > 
> > I've only handled the case of having an SMMU that supports larger
> > s2 input-sizes than what p2m wants to use.
> > 
> > To support the case were the system has SMMUs that can only do smaller
> > input-sizes than the CPU, I think we will need to walk the device-tree
> > when creating domains and somehow cap the p2m IPA size to the largest
> > supported S2 size across the SMMUs used by that particular domain.
> > The max IPA size may need to be domain specific.
> > I did not implement this but I did make the max size domain specific.
> > Julien, you mentioned you might have access to such a system?
> 
> Right, I got an AMD Seattle where the IPA is limited by the SMMU. From
> Xen output:
>    - SMMU: 40-bit IPA -> 40-bit PA
>    - P2M: 44-bit IPA with 44-bit PA
> 
> Suravee, can you confirm that this is actually the case?
> 
> > 
> > Does this look reasonable or should we be fixing this some other way?
> > I could make the patch simpler I guess by having a global (not
> > domain specific) max value set once aswell.
> 
> For your use case a global seems better (you may need to move the
> iommu_setup call after setup_virt_paging).

OK, I'll implement that and we can incrementally add a per-domain
values if needed.

> Although, if the limitation of the SMMU on Seattle is confirmed, that
> would restrict domain memory to 1TB maximum (rather than 16TB) on this
> platform.

Right, with the domain specific max ipa value, we could limit the address
space only for guests that actually use the SMMU. Not sure how much value
there is in that though...

Thanks!

Cheers,
Edgar


> 
> Regards,
> 
> -- 
> Julien Grall