Re: Load increase after memory upgrade (part2)

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Mon, Nov 28, 2011 at 03:40:13PM +0000, Ian Campbell wrote:
> On Mon, 2011-11-28 at 15:28 +0000, Konrad Rzeszutek Wilk wrote:
> > On Fri, Nov 25, 2011 at 11:11:55PM +0100, Carsten Schiers wrote:
> 
> > > I looked through my old mails from you and you explained already the necessity of double
> > > bounce buffering (PCI->below 4GB->above 4GB). What I don't understand is: why does the
> > > Xenified kernel not have this kind of issue?
> > 
> > That is a puzzle. It should not. The code is very much the same - both
> > use the generic SWIOTLB which has not changed for years.
> 
> The swiotlb-xen used by classic-xen kernels (which I assume is what
> Carsten means by "Xenified") isn't exactly the same as the stuff in
> mainline Linux, it's been heavily refactored for one thing. It's not
> impossible that mainline is bouncing something it doesn't really need
> to.

The usage, at least with 'pci_alloc_coherent' is that there is no bouncing
being done. The alloc_coherent will allocate a nice page, underneath the 4GB
mark and give it to the driver. The driver can use it as it wishes and there
is no need to bounce buffer.

But I can't find the implementation of that in the classic Xen-SWIOTLB. It looks
as if it is using map_single which would be taking the memory out of the
pool for a very long time, instead of allocating memory and "swizzling" the MFNs.
[Note, I looked at the 2.6.18 hg tree for classic, the 2.6.34 is probably
improved much better so let me check that]

Carsten, let me prep up a patch that will print some diagnostic information
during the runtime - to see how often it does the bounce, the usage, etc..

> 
> It's also possible that the dma mask of the device is different/wrong in
> mainline leading to such additional bouncing.

If one were to use map_page and such - yes. But the alloc_coherent bypasses
that and ends up allocating it right under the 4GB (or rather it allocates
based on the dev->coherent_mask and swizzles the MFNs as required).

> 
> I guess it's also possible that the classic-Xen kernels are playing fast
> and loose by not bouncing something they should (although if so they
> appear to be getting away with it...) or that there is some difference
> which really means mainline needs to bounce while classic-Xen doesn't.

<nods> Could be very well.
> 
> Ian.
>