Re: PATCH: Further aacraid work

[Andi Kleen <ak@muc.de>]

Anton Blanchard <anton@samba.org> writes:

> Please divert some of your anger towards your manufacturer of dodgy
> hardware. Any sane hardware with an IOMMU handles this just fine.
> eg on ppc64 running a disk test:
>
> sg size    in        out
> 1           3      47569
> 2           0       2591
> 3           0       1123
> 4           0        447
> 5           0        429
> ...
> 62       5095          0
> 64      47061          0
>
> The IOMMU is taking 62-64 entry SG lists and producing 1-5 entry lists.

The AMD64 IOMMU could do it too (and the code to do it exists in
2.6). But the problem is that the current IO layer doesn't provide a
sufficient fallback path when this fails. You have to promise in
advance that you can merge and then later it's too late to change your
mind without signalling an IO error.

This is a real problem on AMD64, because IOMMU aperture is relatively
small and can fragment. 

I had a chat with James about this at last year's OLS. The Consensus
was iirc that it needs driver interface changes at least.

If there was a sane fallback path for this I would enable merging
always (and add some fragmentation avoidance algorithms to the 
bitmap allocator to make failure less likely)

It's also a balancing act in terms of performance. The IOMMU setup
is relatively slow (it has to do an PCI config space write and 
an uncached memory access), and it depends on the device if it's 
actually faster to go through the IOMMU. I did some benchmarks
and it seems to help on MPT Fusion controllers, but slows down
ethernet. Most probably we need an driver function call where
the driver can tell the IOMMU layer "I am slow at merging; 
give me merged sg lists and i can handle errors by falling back"

Also of course when the merging is used you will always get
addresses <32bit and can potentially use smaller descriptors.
But again you need fallback, because on AMD64 the IOMMus 
can be quite small and it's possible to overflow them 
in extreme traffic situations.

-Andi