Re: question about memory allocation for driver domain

[Ian Campbell <Ian.Campbell@citrix.com>]

On Wed, 2015-02-04 at 18:47 +0200, Oleksandr Tyshchenko wrote:
> Hi, all.
> 
> We have begun to use the driver domain on OMAP5 platform.
> To make driver domain running on OMAP5 platform we need to have it
> memory 1 to 1 mapped because of lacking SMMU support on this platform.
> To satisfy this requirement we have done temporally solution which
> works but not entirely completed and looks not good from Xen
> perspective. The main question in memory allocator in Xen.
> 
> We did next steps:
> 1. toolstack:
> - allow to allocate 128/256/512 Mb memory chunks
> - add ability to set rambase_pfn via cfg file
> 
> 2. hypervisor:
> - alloc driver domain memory 1 to 1
>   - mark domain with id 1 as privileged
>   - update memory allocation logic for such domain
>   - allocate memory in the same way as for domain 0
> 
> But we have encountered with one thing related to memory allocation algorithm.
> Let me describe what I mean.
> Our requirements is to allocate one specified chunk if it is present in domheap.
> We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
> want to map 1 to 1. We can get it from extent_list. So, we need to
> alloc chunk which would be correspond to "spec_mfn".
> In other world we need to allocate known chunk of memory. But if I
> understood correctly that the existing allocator doesn't allow us to
> do that directly.
> 
> There are some thoughts how to do that. But, at first, we need to
> choose right direction:
> 1. Add the separate memory allocator which allow us to alloc specified
> chunk if it is present. And use it in case when we need to allocate
> memory for driver domain 1:1 only. We can pass bool variable via cfg
> file to show what we want (1:1 or not).
> 2. Don't add separate allocator. Modify existing allocator to add
> ability to alloc specified chunk.
> 3. Don't add and modify anything. Let the allocator to work as usual.
> Return mfn of allocating chunk and correct the default rambase_pfn in
> toolstack.
> What we actually have in the moment but do that manually. We see what
> the mfn we got and corrected "rambase_pfn" property in cfg file.
> 
> Could someone explain me what is a right way?

The approach we've taken with dom0 is not to require specific addresses,
but rather to tailor the address map to the contiguous region the
allocator gives us.

Since I presume you are using a 1:1 layout in the driver domain too I
expect that approach should work there too (I think this is your #3?).

Your #2 might be possible, but would probably involve a reasonably
expensive scan of the various free pools in the hopes of finding the
block you want, since it isn't designed to be looked up in this way.

I suppose #1 would be something like Linux's CMA allocator -- i.e.
carving out 1 or more regions on boot and keeping them away from the
main allocator (but still in e.g. the frametable etc) and a simple way
to allocate one of the chunks.

So I think any of the approaches you suggest could work, I'd probably
say #3, #1, #2 in decreasing order of preference.

Now, if you were asking for ideas on how to make this stuff
upstreamable, well that's a harder question. I'm not really sure :-/

Ian.