Re: [was Re: Linux 2.6.14 ] Revert "x86-64: Avoid unnecessary double bouncing for swiotlb"

[Andi Kleen <ak@suse.de>]

On Saturday 29 October 2005 00:58, Ravikiran G Thirumalai wrote:
> On Thu, Oct 27, 2005 at 05:28:50PM -0700, Linus Torvalds wrote:
> >       Revert "x86-64: Avoid unnecessary double bouncing for swiotlb"
>
> (http://www.kernel.org/git/?p=linux/kernel/git/torvalds/linux-2.6.git;a=com
>mitdiff;h=79b95a454bb5c1d9b7287d1016a70885ba3f346c)
>
> Well, Andi's patch here wasn't just a small optimization as the changelog
> suggests. It helped EM64T boxes a great deal.  \

First to be honest swiotlb performance is not very high on the priority list. 
It will always be bad. If you care about performance you should use devices 
that can address all your memory.

EM64T server boxes shouldn't have big problems with that because they usually
support AHCI for IDE, and firewire/usb2/sound is not that critical. And the 
EM64T boxes with other chipsets typically don't support >4G phys because they 
only support the lowerend Intel CPUs. Summit might be an exception, but those
normally only use IDE for CDROMs, which are also not a big issue.

> Just to make sure, I 
> reran 2.6.14 with the attached patch and got about 45% better performance
> with iozone Initial write.  This was on a 2 cpu 4 thread SMP Xeon with 8G
> ram, with 2 processes performing io to 4G files on a IDE drive.
> Maybe it wouldn't have caused breakage on some AMD boxes if the following
> additional check for swiotlb was added.  Can this go into 2.6.15 please?

Not in this form no. Problem first needs to be understood fully and
then no_iommu should be set properly.

>   * On AMD64 it mostly equals, but we set it to zero to tell some
> subsystems - * that an IOMMU is available.
> + * that a hard or soft IOMMU is available.
>   */
> -#define PCI_DMA_BUS_IS_PHYS	(no_iommu ? 1 : 0)
> +#define PCI_DMA_BUS_IS_PHYS	((no_iommu && !swiotlb) ? 1 : 0)

That is ugly and I don't like it.  Need to track down the real problem 

-Andi