Re: Assigning contiguous memory to a driver domain

[Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>]

On Wed, Sep 15, 2010 at 04:29:28PM +0100, Jan Beulich wrote:
> >>> On 15.09.10 at 16:44, Rafal Wojtczuk <rafal@invisiblethingslab.com> wrote:
> > On Wed, Sep 15, 2010 at 02:49:37PM +0100, Jan Beulich wrote:
> >> Because on suspend the driver frees the memory which on resume
> >> it will allocate back?
> > I am a bit lost.
> > By "frees the memory" you mean "return contiguous memory to Xen free memory" 
> > ?
> > Does it really work this way ?
> 
> Yes - the "special" memory gets exchanged back to "normal" memory
> upon freeing of it by the driver. The exception is if Xen has no "normal"
> memory left to give back out in exchange - in that case the domain will
> retain the "special" memory indefinitely. Yes, you can call this a leak,
> but no, I don't think there's much you can do about it (without adding
> likely rather complex extra code).

Let me expand this. During bootup Xen-SWIOTLB  (which for DomU you have
to enable via the 'iommu=soft'), allocated 32 2MB chunks of contingous
memory under the 4GB limit. Those chunks stay in DomU and are used
during the the runtime of the DomU. They don't go back to Xen unless the
domain has been terminated. Any of the DMA operations that any driver
does go through the SWIOTLB bufer if the physical (mfn) for the DMA is
outside the driver capabilities (say, your ping buffer is allocated above
the 4GB, and your r8169 can only do 32-bit, then SWIOTLB would be utilized
to "bounce" the memory).


> 
> > If so, it requires nonzero Xen free memory ? And that is why when I do
> > "ifconfig eth0 down; ifconfig eth0 up" in the driver domain the second one
> > fails ?

There are couple of things happening when you do ifconfig eth0 up. The 
PTE used for virtual address for the BARs are updated with the _PAGE_IOMAP
which means that the GMFN->PFN->MFN is shortcircuited to be GMFN->MFN.
Obviously that doesn' use any Xen heap memory. The next thing is
that the the driver might allocate coherent DMA mappings. Those are the
ones I think Jan is referring to. For coherent DMA mappings we just
do page_alloc and then we swap the memory behind those pages with Xen
to be under the 32-bit limit (xen_create_contiguous_region).
Naturally when the driver is unloaded the de-allocation will call
xen_destroy_contiguous_region. Loking at the code I think it swaps with
the highest bit order (so with memory close to the end of physical space).


> 
> Generally the second "up" shouldn't fail as long as the prior "down"
> properly returned all resources. See the restrictions above.

Yeah, it might be worth looking at what it is doing to cause this. The e1000/igb
are pretty good at cleaning everying so you can do ifup;ifdown indefinitly.

In reference to the Xen-SWIOTLB for other versions that upstream, there are a couple
of implementations at:
git://git.kernel.org/pub/scm/linux/kernel/git/konrad/swiotlb-2.6.git

for different Linux versions. Which version of kernel do you guys use?
> 
> Jan
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel