RE: A problem of Intel IOMMU hardware ？ - Longpeng (Mike, Cloud Infrastructure Service Product Dept.)

From: "Longpeng (Mike, Cloud Infrastructure Service Product Dept.)" <longpeng2@huawei.com>
To: Nadav Amit <nadav.amit@gmail.com>
Cc: chenjiashang <chenjiashang@huawei.com>,
	David Woodhouse <dwmw2@infradead.org>,
	"iommu@lists.linux-foundation.org"
	<iommu@lists.linux-foundation.org>,
	LKML <linux-kernel@vger.kernel.org>,
	"alex.williamson@redhat.com" <alex.williamson@redhat.com>,
	"Gonglei \(Arei\)" <arei.gonglei@huawei.com>,
	"will@kernel.org" <will@kernel.org>
Subject: RE: A problem of Intel IOMMU hardware ？
Date: Wed, 17 Mar 2021 09:35:51 +0000	[thread overview]
Message-ID: <6a218e7fe42d41489d02f0b4e3ad2756@huawei.com> (raw)
In-Reply-To: <FB4E11A5-84D4-4DAF-889E-FAA1BCD2E66F@gmail.com>

Hi Nadav,

> -----Original Message-----
> From: Nadav Amit [mailto:nadav.amit@gmail.com]
> Sent: Wednesday, March 17, 2021 1:46 PM
> To: Longpeng (Mike, Cloud Infrastructure Service Product Dept.)
> <longpeng2@huawei.com>
> Cc: David Woodhouse <dwmw2@infradead.org>; Lu Baolu
> <baolu.lu@linux.intel.com>; Joerg Roedel <joro@8bytes.org>; will@kernel.org;
> alex.williamson@redhat.com; chenjiashang <chenjiashang@huawei.com>;
> iommu@lists.linux-foundation.org; Gonglei (Arei) <arei.gonglei@huawei.com>;
> LKML <linux-kernel@vger.kernel.org>
> Subject: Re: A problem of Intel IOMMU hardware ？
> 
> 
> 
> > On Mar 16, 2021, at 8:16 PM, Longpeng (Mike, Cloud Infrastructure Service
> Product Dept.) <longpeng2@huawei.com> wrote:
> >
> > Hi guys,
> >
> > We find the Intel iommu cache (i.e. iotlb) maybe works wrong in a
> > special situation, it would cause DMA fails or get wrong data.
> >
> > The reproducer (based on Alex's vfio testsuite[1]) is in attachment,
> > it can reproduce the problem with high probability (~50%).
> 
> I saw Lu replied, and he is much more knowledgable than I am (I was just intrigued
> by your email).
> 
> However, if I were you I would try also to remove some “optimizations” to look for
> the root-cause (e.g., use domain specific invalidations instead of page-specific).
> 

Good suggestion! But we did it these days, we tried to use global invalidations as follow:
		iommu->flush.flush_iotlb(iommu, did, 0, 0,
						DMA_TLB_DSI_FLUSH);
But can not resolve the problem.

> The first thing that comes to my mind is the invalidation hint (ih) in
> iommu_flush_iotlb_psi(). I would remove it to see whether you get the failure
> without it.

We also notice the IH, but the IH is always ZERO in our case, as the spec says:
'''
Paging-structure-cache entries caching second-level mappings associated with the specified
domain-id and the second-level-input-address range are invalidated, if the Invalidation Hint
(IH) field is Clear.
'''

It seems the software is everything fine, so we've no choice but to suspect the hardware.
_______________________________________________
iommu mailing list
iommu@lists.linux-foundation.org
https://lists.linuxfoundation.org/mailman/listinfo/iommu