Re: IOMMUs was Re: Intel vs AMD x86-64

[Benjamin Herrenschmidt <benh@kernel.crashing.org>]

> Arjan suggested it some time ago already. In fact I implemented it, it's in the current code.
> But it caused data corruption with a few devices, in particular 3ware, so I had 
> to disable it again. I didn't find a bug in the code. It worked fine with others. My theory 
> was that it triggered some hardware bug that was normally masked by the frequent flushes, but 
> I wasn't able to track it down without heavy equipment.

Interesting. I'm having a data corruption issue with the G5 iommu that
I can fix by always mapping everything. That is non-mapped virtual
IO pages are actually mapped to a dummy RAM page. It seems there is a
problem with the PCI<->HT bridge doing prefetches beyond iommu mapped
pages, thus triggering an iommu error, which in turns probably triggers
some other chipset bug ending up in data corruption. Having everything
mapped (allowing prefetch to complete even while prefetched data is
actually useless) fixes the problem and we don't see any corruption.

Of course, that means we can not longer use the mecanism we first
implemented where we would only flush the iommu TLB once after runnning
out of virtual pages to allocate. We have to flush on every insertion
and removal now :( On the other hand, we can probably do per-tag TLB
flushes instead of flushing the whole TLB once we properly figure out
how to access the tag registers on the chipset and their format (the
darwin source code seem to imply that is doable, but doesn't actually
use that, but in this regard, apple's implementation is impressively
sub-optimal).

Ben.