Re: [BUGREPORT] Intel X58 Boards don't boot after "iommu/vt-d: Make root entry visible for hardware right after allocation"

[Joerg Roedel <joro-zLv9SwRftAIdnm+yROfE0A@public.gmane.org>]

On Thu, Jun 16, 2016 at 10:26:20AM +0400, Dheeraj CVR wrote:
> I have added some debugging code to see where exactly the hang happens. The 
> set_root_entry call works fine without any issues as you have expected.
> However, the hang happens when calling flush_context.
> 
> flush_context -> qi_flush_context -> qi_submit_sync
> 
> The loop "while (qi->desc_status[wait_index] != QI_DONE) {}" in
> "qi_submit_sync" executes indefinitely and the control doesn't break. Hence, "
> iommu->flush.flush_context()" never returns.
> 
> Tested on 2 X58 Motherboards and both hang at the same place with intel_iommu=
> on.
> 
> Hope this helps.

Hmm, okay. Yesterday I tried to reproduce the problem on a box in our
test-lab which has the same chipset. The bug didn't show up there, and
one difference to your machine is that is only has one iommu instead of
two.

This brought me to another difference the patch made: Before the patch
QI was enabled on all iommus before the root entry was set and the
caches flushed. With the patch QI is enabled in the same loop that sets
the root entry and tries to flush. Maybe that is the problem on this
chipset.

Can you please try the attached diff? It is just a guess, but maybe that
is the problem.

Thanks,

	Joerg

diff --git a/drivers/iommu/intel-iommu.c b/drivers/iommu/intel-iommu.c
index a644d0c..0ddbb6a 100644
--- a/drivers/iommu/intel-iommu.c
+++ b/drivers/iommu/intel-iommu.c
@@ -3222,11 +3222,6 @@ static int __init init_dmars(void)
 			}
 		}
 
-		iommu_flush_write_buffer(iommu);
-		iommu_set_root_entry(iommu);
-		iommu->flush.flush_context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL);
-		iommu->flush.flush_iotlb(iommu, 0, 0, 0, DMA_TLB_GLOBAL_FLUSH);
-
 		if (!ecap_pass_through(iommu->ecap))
 			hw_pass_through = 0;
 #ifdef CONFIG_INTEL_IOMMU_SVM
@@ -3235,6 +3230,17 @@ static int __init init_dmars(void)
 #endif
 	}
 
+	/*
+	 * Now that qi is enabled on all iommus, set the root entry and flush
+	 * caches.
+	 */
+	for_each_active_iommu(iommu, drhd) {
+		iommu_flush_write_buffer(iommu);
+		iommu_set_root_entry(iommu);
+		iommu->flush.flush_context(iommu, 0, 0, 0, DMA_CCMD_GLOBAL_INVL);
+		iommu->flush.flush_iotlb(iommu, 0, 0, 0, DMA_TLB_GLOBAL_FLUSH);
+	}
+
 	if (iommu_pass_through)
 		iommu_identity_mapping |= IDENTMAP_ALL;