Re: [PATCH 2/3] x86-64: Calgary IOMMU - Calgary specific bits

[Andi Kleen <ak@suse.de>]

On Thursday 23 March 2006 18:53, Muli Ben-Yehuda wrote:

> no we don't, would you prefer we do it there?

Yes

> > > +	/* make sure updates are seen by hardware */
> > > +	mb();
> > 
> > Doesn't make sense on x86-64.
> 
> would you mind explaining why? we need the tce update to be flushed
> out of the cache to main memory, is this implied by something that
> happens earlier, e.g. the spin_unlock?

Barriers don't have anything to do with flushing. See the long thread
I had recently about this with the ipath person.

And on x86-64 the writes are always ordered so mb() normally doesn't
affect writes at all (except for compiler reordering but I can't see
this being a problem here)

If you want a real flush you need CLFLUSH if it's cached and
a read if it's behind a PCI bridge.

> the way the chip works is that incoming addresses from the device that
> land inside these holes will not get translated, so we have to make
> sure not to give them out to devices. AFAIK these are the only holes
> in the IO space as far as the chip is concerned. At least empirically
> it works :-)

The 640K-1MB is in no way different from the PCI hole below 4GB in this regard.

It's still totally unclear why you special case one and not the other.

In general you should probably drive this over e820 and only allow RAM
(or hotplug memory beyond end_pfn). Or not have this special case at all.



> we ran into that; we use the bus's sysdata for NUMA, whereas here we
> use the bus's pci_dev's sysdata, so there isn't any conflict. If you
> prefer that we allocate a new structure and use that for both NUMA and
> Calgary, we can certainly do it.

Ok but needs prominent comments.

> it should, but at the moment swiotlb is tied too intimately to
> gart and doesn't work with anything else. It's on our TODO list.


I don't quite buy this. With the gart ops this really shouldn't 
be a problem anymore.


> > I would like to see a printk and some comments about the full isolation
> > because it's a big change. How does it interact with the X server
> > for once?
> 
> X works :-) 

So it's behind a bridge that doesn't have an IOMMU?
Ok i suppose dual head will not work but you might not care.

Otherwise you would need to add an interface to disable the isolation
from user space for X.

-Andi