question about memory allocation for driver domain

question about memory allocation for driver domain

[Oleksandr Tyshchenko]

Hi, all.

We have begun to use the driver domain on OMAP5 platform.
To make driver domain running on OMAP5 platform we need to have it
memory 1 to 1 mapped because of lacking SMMU support on this platform.
To satisfy this requirement we have done temporally solution which
works but not entirely completed and looks not good from Xen
perspective. The main question in memory allocator in Xen.

We did next steps:
1. toolstack:
- allow to allocate 128/256/512 Mb memory chunks
- add ability to set rambase_pfn via cfg file

2. hypervisor:
- alloc driver domain memory 1 to 1
  - mark domain with id 1 as privileged
  - update memory allocation logic for such domain
  - allocate memory in the same way as for domain 0

But we have encountered with one thing related to memory allocation algorithm.
Let me describe what I mean.
Our requirements is to allocate one specified chunk if it is present in domheap.
We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
want to map 1 to 1. We can get it from extent_list. So, we need to
alloc chunk which would be correspond to "spec_mfn".
In other world we need to allocate known chunk of memory. But if I
understood correctly that the existing allocator doesn't allow us to
do that directly.

There are some thoughts how to do that. But, at first, we need to
choose right direction:
1. Add the separate memory allocator which allow us to alloc specified
chunk if it is present. And use it in case when we need to allocate
memory for driver domain 1:1 only. We can pass bool variable via cfg
file to show what we want (1:1 or not).
2. Don't add separate allocator. Modify existing allocator to add
ability to alloc specified chunk.
3. Don't add and modify anything. Let the allocator to work as usual.
Return mfn of allocating chunk and correct the default rambase_pfn in
toolstack.
What we actually have in the moment but do that manually. We see what
the mfn we got and corrected "rambase_pfn" property in cfg file.

Could someone explain me what is a right way?
All suggestions welcomed!

-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

Re: question about memory allocation for driver domain

[Ian Campbell]

On Wed, 2015-02-04 at 18:47 +0200, Oleksandr Tyshchenko wrote:
> Hi, all.
> 
> We have begun to use the driver domain on OMAP5 platform.
> To make driver domain running on OMAP5 platform we need to have it
> memory 1 to 1 mapped because of lacking SMMU support on this platform.
> To satisfy this requirement we have done temporally solution which
> works but not entirely completed and looks not good from Xen
> perspective. The main question in memory allocator in Xen.
> 
> We did next steps:
> 1. toolstack:
> - allow to allocate 128/256/512 Mb memory chunks
> - add ability to set rambase_pfn via cfg file
> 
> 2. hypervisor:
> - alloc driver domain memory 1 to 1
>   - mark domain with id 1 as privileged
>   - update memory allocation logic for such domain
>   - allocate memory in the same way as for domain 0
> 
> But we have encountered with one thing related to memory allocation algorithm.
> Let me describe what I mean.
> Our requirements is to allocate one specified chunk if it is present in domheap.
> We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
> want to map 1 to 1. We can get it from extent_list. So, we need to
> alloc chunk which would be correspond to "spec_mfn".
> In other world we need to allocate known chunk of memory. But if I
> understood correctly that the existing allocator doesn't allow us to
> do that directly.
> 
> There are some thoughts how to do that. But, at first, we need to
> choose right direction:
> 1. Add the separate memory allocator which allow us to alloc specified
> chunk if it is present. And use it in case when we need to allocate
> memory for driver domain 1:1 only. We can pass bool variable via cfg
> file to show what we want (1:1 or not).
> 2. Don't add separate allocator. Modify existing allocator to add
> ability to alloc specified chunk.
> 3. Don't add and modify anything. Let the allocator to work as usual.
> Return mfn of allocating chunk and correct the default rambase_pfn in
> toolstack.
> What we actually have in the moment but do that manually. We see what
> the mfn we got and corrected "rambase_pfn" property in cfg file.
> 
> Could someone explain me what is a right way?

The approach we've taken with dom0 is not to require specific addresses,
but rather to tailor the address map to the contiguous region the
allocator gives us.

Since I presume you are using a 1:1 layout in the driver domain too I
expect that approach should work there too (I think this is your #3?).

Your #2 might be possible, but would probably involve a reasonably
expensive scan of the various free pools in the hopes of finding the
block you want, since it isn't designed to be looked up in this way.

I suppose #1 would be something like Linux's CMA allocator -- i.e.
carving out 1 or more regions on boot and keeping them away from the
main allocator (but still in e.g. the frametable etc) and a simple way
to allocate one of the chunks.

So I think any of the approaches you suggest could work, I'd probably
say #3, #1, #2 in decreasing order of preference.

Now, if you were asking for ideas on how to make this stuff
upstreamable, well that's a harder question. I'm not really sure :-/

Ian.

Re: question about memory allocation for driver domain

[Oleksandr Tyshchenko]

Hi, Ian

On Thu, Feb 5, 2015 at 3:12 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
> On Wed, 2015-02-04 at 18:47 +0200, Oleksandr Tyshchenko wrote:
>> Hi, all.
>>
>> We have begun to use the driver domain on OMAP5 platform.
>> To make driver domain running on OMAP5 platform we need to have it
>> memory 1 to 1 mapped because of lacking SMMU support on this platform.
>> To satisfy this requirement we have done temporally solution which
>> works but not entirely completed and looks not good from Xen
>> perspective. The main question in memory allocator in Xen.
>>
>> We did next steps:
>> 1. toolstack:
>> - allow to allocate 128/256/512 Mb memory chunks
>> - add ability to set rambase_pfn via cfg file
>>
>> 2. hypervisor:
>> - alloc driver domain memory 1 to 1
>>   - mark domain with id 1 as privileged
>>   - update memory allocation logic for such domain
>>   - allocate memory in the same way as for domain 0
>>
>> But we have encountered with one thing related to memory allocation algorithm.
>> Let me describe what I mean.
>> Our requirements is to allocate one specified chunk if it is present in domheap.
>> We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
>> want to map 1 to 1. We can get it from extent_list. So, we need to
>> alloc chunk which would be correspond to "spec_mfn".
>> In other world we need to allocate known chunk of memory. But if I
>> understood correctly that the existing allocator doesn't allow us to
>> do that directly.
>>
>> There are some thoughts how to do that. But, at first, we need to
>> choose right direction:
>> 1. Add the separate memory allocator which allow us to alloc specified
>> chunk if it is present. And use it in case when we need to allocate
>> memory for driver domain 1:1 only. We can pass bool variable via cfg
>> file to show what we want (1:1 or not).
>> 2. Don't add separate allocator. Modify existing allocator to add
>> ability to alloc specified chunk.
>> 3. Don't add and modify anything. Let the allocator to work as usual.
>> Return mfn of allocating chunk and correct the default rambase_pfn in
>> toolstack.
>> What we actually have in the moment but do that manually. We see what
>> the mfn we got and corrected "rambase_pfn" property in cfg file.
>>
>> Could someone explain me what is a right way?
>
> The approach we've taken with dom0 is not to require specific addresses,
> but rather to tailor the address map to the contiguous region the
> allocator gives us.
>
> Since I presume you are using a 1:1 layout in the driver domain too I
> expect that approach should work there too (I think this is your #3?).
yes
>
> Your #2 might be possible, but would probably involve a reasonably
> expensive scan of the various free pools in the hopes of finding the
> block you want, since it isn't designed to be looked up in this way.
>
> I suppose #1 would be something like Linux's CMA allocator -- i.e.
> carving out 1 or more regions on boot and keeping them away from the
> main allocator (but still in e.g. the frametable etc) and a simple way
> to allocate one of the chunks.
This is interesting. Can I use /memreserve/ logic or something similar to
keeping them away from the main allocator?
But at the stages where we "initialize" domheap we know nothing about
guest domain (domd) and how we need to allocate memory for it (1:1 or not).

>
> So I think any of the approaches you suggest could work, I'd probably
> say #3, #1, #2 in decreasing order of preference.
I got it.

>
> Now, if you were asking for ideas on how to make this stuff
> upstreamable, well that's a harder question. I'm not really sure :-/
Ideally, I would like to make this stuff upstreamable.

>
> Ian.
>



-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

Re: question about memory allocation for driver domain

[Julien Grall]

Hi Oleksandr,

On 05/02/2015 21:49, Oleksandr Tyshchenko wrote:
> On Thu, Feb 5, 2015 at 3:12 PM, Ian Campbell <Ian.Campbell@citrix.com> wrote:
>> On Wed, 2015-02-04 at 18:47 +0200, Oleksandr Tyshchenko wrote:
>>> Hi, all.
>>>
>>> We have begun to use the driver domain on OMAP5 platform.
>>> To make driver domain running on OMAP5 platform we need to have it
>>> memory 1 to 1 mapped because of lacking SMMU support on this platform.
>>> To satisfy this requirement we have done temporally solution which
>>> works but not entirely completed and looks not good from Xen
>>> perspective. The main question in memory allocator in Xen.
>>>
>>> We did next steps:
>>> 1. toolstack:
>>> - allow to allocate 128/256/512 Mb memory chunks
>>> - add ability to set rambase_pfn via cfg file
>>>
>>> 2. hypervisor:
>>> - alloc driver domain memory 1 to 1
>>>    - mark domain with id 1 as privileged
>>>    - update memory allocation logic for such domain
>>>    - allocate memory in the same way as for domain 0
>>>
>>> But we have encountered with one thing related to memory allocation algorithm.
>>> Let me describe what I mean.
>>> Our requirements is to allocate one specified chunk if it is present in domheap.
>>> We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
>>> want to map 1 to 1. We can get it from extent_list. So, we need to
>>> alloc chunk which would be correspond to "spec_mfn".
>>> In other world we need to allocate known chunk of memory. But if I
>>> understood correctly that the existing allocator doesn't allow us to
>>> do that directly.
>>>
>>> There are some thoughts how to do that. But, at first, we need to
>>> choose right direction:
>>> 1. Add the separate memory allocator which allow us to alloc specified
>>> chunk if it is present. And use it in case when we need to allocate
>>> memory for driver domain 1:1 only. We can pass bool variable via cfg
>>> file to show what we want (1:1 or not).
>>> 2. Don't add separate allocator. Modify existing allocator to add
>>> ability to alloc specified chunk.
>>> 3. Don't add and modify anything. Let the allocator to work as usual.
>>> Return mfn of allocating chunk and correct the default rambase_pfn in
>>> toolstack.
>>> What we actually have in the moment but do that manually. We see what
>>> the mfn we got and corrected "rambase_pfn" property in cfg file.
>>>
>>> Could someone explain me what is a right way?
>>
>> The approach we've taken with dom0 is not to require specific addresses,
>> but rather to tailor the address map to the contiguous region the
>> allocator gives us.
>>
>> Since I presume you are using a 1:1 layout in the driver domain too I
>> expect that approach should work there too (I think this is your #3?).
> yes
>>
>> Your #2 might be possible, but would probably involve a reasonably
>> expensive scan of the various free pools in the hopes of finding the
>> block you want, since it isn't designed to be looked up in this way.
>>
>> I suppose #1 would be something like Linux's CMA allocator -- i.e.
>> carving out 1 or more regions on boot and keeping them away from the
>> main allocator (but still in e.g. the frametable etc) and a simple way
>> to allocate one of the chunks.
> This is interesting. Can I use /memreserve/ logic or something similar to
> keeping them away from the main allocator?
> But at the stages where we "initialize" domheap we know nothing about
> guest domain (domd) and how we need to allocate memory for it (1:1 or not).

The first approach we had for DOM0 1:1 mapping was book-keeping memory 
at startup. The main allocator wasn't able to use it.

But this turn out to be very hackish and not scalable (multiple memory 
banks...).

>>
>> So I think any of the approaches you suggest could work, I'd probably
>> say #3, #1, #2 in decreasing order of preference.
> I got it.

If I correctly understand the solution #3, it may end up to lots of 
small banks after domains has been created/destroyed multiple

>>
>> Now, if you were asking for ideas on how to make this stuff
>> upstreamable, well that's a harder question. I'm not really sure :-/
> Ideally, I would like to make this stuff upstreamable.

As you said on your first mail, this is required to allow DMA on device 
passthrough when the platform doesn't have an SMMU.

I totally understand a such use case for specific embedded product, 
because you trust your driver domains.

On upstream, the main use case of driver domain is too protected the 
platform from buggy drivers. If the drivers crash, then you can restart 
only this domain.

With a device protected by an SMMU, any DMA requests are safe. Without 
it, the driver can pretty much do whatever it wants. This could lead to 
the hypervisor corruption/crash.

If a such feature as to come to Xen upstream, this should be hidden to 
the main users (maybe via compilation option) or require a specific 
option to use it.

Regards.

-- 
Julien Grall

Re: question about memory allocation for driver domain

[Oleksandr Tyshchenko]

On Thu, Feb 5, 2015 at 5:24 PM, Julien Grall <julien.grall@linaro.org> wrote:
> Hi Oleksandr,
Hi Julien
>
>
> On 05/02/2015 21:49, Oleksandr Tyshchenko wrote:
>>
>> On Thu, Feb 5, 2015 at 3:12 PM, Ian Campbell <Ian.Campbell@citrix.com>
>> wrote:
>>>
>>> On Wed, 2015-02-04 at 18:47 +0200, Oleksandr Tyshchenko wrote:
>>>>
>>>> Hi, all.
>>>>
>>>> We have begun to use the driver domain on OMAP5 platform.
>>>> To make driver domain running on OMAP5 platform we need to have it
>>>> memory 1 to 1 mapped because of lacking SMMU support on this platform.
>>>> To satisfy this requirement we have done temporally solution which
>>>> works but not entirely completed and looks not good from Xen
>>>> perspective. The main question in memory allocator in Xen.
>>>>
>>>> We did next steps:
>>>> 1. toolstack:
>>>> - allow to allocate 128/256/512 Mb memory chunks
>>>> - add ability to set rambase_pfn via cfg file
>>>>
>>>> 2. hypervisor:
>>>> - alloc driver domain memory 1 to 1
>>>>    - mark domain with id 1 as privileged
>>>>    - update memory allocation logic for such domain
>>>>    - allocate memory in the same way as for domain 0
>>>>
>>>> But we have encountered with one thing related to memory allocation
>>>> algorithm.
>>>> Let me describe what I mean.
>>>> Our requirements is to allocate one specified chunk if it is present in
>>>> domheap.
>>>> We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
>>>> want to map 1 to 1. We can get it from extent_list. So, we need to
>>>> alloc chunk which would be correspond to "spec_mfn".
>>>> In other world we need to allocate known chunk of memory. But if I
>>>> understood correctly that the existing allocator doesn't allow us to
>>>> do that directly.
>>>>
>>>> There are some thoughts how to do that. But, at first, we need to
>>>> choose right direction:
>>>> 1. Add the separate memory allocator which allow us to alloc specified
>>>> chunk if it is present. And use it in case when we need to allocate
>>>> memory for driver domain 1:1 only. We can pass bool variable via cfg
>>>> file to show what we want (1:1 or not).
>>>> 2. Don't add separate allocator. Modify existing allocator to add
>>>> ability to alloc specified chunk.
>>>> 3. Don't add and modify anything. Let the allocator to work as usual.
>>>> Return mfn of allocating chunk and correct the default rambase_pfn in
>>>> toolstack.
>>>> What we actually have in the moment but do that manually. We see what
>>>> the mfn we got and corrected "rambase_pfn" property in cfg file.
>>>>
>>>> Could someone explain me what is a right way?
>>>
>>>
>>> The approach we've taken with dom0 is not to require specific addresses,
>>> but rather to tailor the address map to the contiguous region the
>>> allocator gives us.
>>>
>>> Since I presume you are using a 1:1 layout in the driver domain too I
>>> expect that approach should work there too (I think this is your #3?).
>>
>> yes
>>>
>>>
>>> Your #2 might be possible, but would probably involve a reasonably
>>> expensive scan of the various free pools in the hopes of finding the
>>> block you want, since it isn't designed to be looked up in this way.
>>>
>>> I suppose #1 would be something like Linux's CMA allocator -- i.e.
>>> carving out 1 or more regions on boot and keeping them away from the
>>> main allocator (but still in e.g. the frametable etc) and a simple way
>>> to allocate one of the chunks.
>>
>> This is interesting. Can I use /memreserve/ logic or something similar to
>> keeping them away from the main allocator?
>> But at the stages where we "initialize" domheap we know nothing about
>> guest domain (domd) and how we need to allocate memory for it (1:1 or
>> not).
>
>
> The first approach we had for DOM0 1:1 mapping was book-keeping memory at
> startup. The main allocator wasn't able to use it.
>
> But this turn out to be very hackish and not scalable (multiple memory
> banks...).
>
>>>
>>> So I think any of the approaches you suggest could work, I'd probably
>>> say #3, #1, #2 in decreasing order of preference.
>>
>> I got it.
>
>
> If I correctly understand the solution #3, it may end up to lots of small
> banks after domains has been created/destroyed multiple
I do not quite understand why this can happen. Anyway, I need to check
this assumption.
I will let you know about result.

>
>>>
>>> Now, if you were asking for ideas on how to make this stuff
>>> upstreamable, well that's a harder question. I'm not really sure :-/
>>
>> Ideally, I would like to make this stuff upstreamable.
>
>
> As you said on your first mail, this is required to allow DMA on device
> passthrough when the platform doesn't have an SMMU.
>
> I totally understand a such use case for specific embedded product, because
> you trust your driver domains.
>
> On upstream, the main use case of driver domain is too protected the
> platform from buggy drivers. If the drivers crash, then you can restart only
> this domain.
>
> With a device protected by an SMMU, any DMA requests are safe. Without it,
> the driver can pretty much do whatever it wants. This could lead to the
> hypervisor corruption/crash.
>
> If a such feature as to come to Xen upstream, this should be hidden to the
> main users (maybe via compilation option) or require a specific option to
> use it.
Completely agree

>
> Regards.
>
> --
> Julien Grall



-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

Re: question about memory allocation for driver domain

[Oleksandr Tyshchenko]

Hi Julien

On Thu, Feb 5, 2015 at 6:36 PM, Oleksandr Tyshchenko
<oleksandr.tyshchenko@globallogic.com> wrote:
> On Thu, Feb 5, 2015 at 5:24 PM, Julien Grall <julien.grall@linaro.org> wrote:
>> Hi Oleksandr,
> Hi Julien
>>
>>
>> On 05/02/2015 21:49, Oleksandr Tyshchenko wrote:
>>>
>>> On Thu, Feb 5, 2015 at 3:12 PM, Ian Campbell <Ian.Campbell@citrix.com>
>>> wrote:
>>>>
>>>> On Wed, 2015-02-04 at 18:47 +0200, Oleksandr Tyshchenko wrote:
>>>>>
>>>>> Hi, all.
>>>>>
>>>>> We have begun to use the driver domain on OMAP5 platform.
>>>>> To make driver domain running on OMAP5 platform we need to have it
>>>>> memory 1 to 1 mapped because of lacking SMMU support on this platform.
>>>>> To satisfy this requirement we have done temporally solution which
>>>>> works but not entirely completed and looks not good from Xen
>>>>> perspective. The main question in memory allocator in Xen.
>>>>>
>>>>> We did next steps:
>>>>> 1. toolstack:
>>>>> - allow to allocate 128/256/512 Mb memory chunks
>>>>> - add ability to set rambase_pfn via cfg file
>>>>>
>>>>> 2. hypervisor:
>>>>> - alloc driver domain memory 1 to 1
>>>>>    - mark domain with id 1 as privileged
>>>>>    - update memory allocation logic for such domain
>>>>>    - allocate memory in the same way as for domain 0
>>>>>
>>>>> But we have encountered with one thing related to memory allocation
>>>>> algorithm.
>>>>> Let me describe what I mean.
>>>>> Our requirements is to allocate one specified chunk if it is present in
>>>>> domheap.
>>>>> We have this "spec_mfn" which are a "rambase_pfn" for driver domain we
>>>>> want to map 1 to 1. We can get it from extent_list. So, we need to
>>>>> alloc chunk which would be correspond to "spec_mfn".
>>>>> In other world we need to allocate known chunk of memory. But if I
>>>>> understood correctly that the existing allocator doesn't allow us to
>>>>> do that directly.
>>>>>
>>>>> There are some thoughts how to do that. But, at first, we need to
>>>>> choose right direction:
>>>>> 1. Add the separate memory allocator which allow us to alloc specified
>>>>> chunk if it is present. And use it in case when we need to allocate
>>>>> memory for driver domain 1:1 only. We can pass bool variable via cfg
>>>>> file to show what we want (1:1 or not).
>>>>> 2. Don't add separate allocator. Modify existing allocator to add
>>>>> ability to alloc specified chunk.
>>>>> 3. Don't add and modify anything. Let the allocator to work as usual.
>>>>> Return mfn of allocating chunk and correct the default rambase_pfn in
>>>>> toolstack.
>>>>> What we actually have in the moment but do that manually. We see what
>>>>> the mfn we got and corrected "rambase_pfn" property in cfg file.
>>>>>
>>>>> Could someone explain me what is a right way?
>>>>
>>>>
>>>> The approach we've taken with dom0 is not to require specific addresses,
>>>> but rather to tailor the address map to the contiguous region the
>>>> allocator gives us.
>>>>
>>>> Since I presume you are using a 1:1 layout in the driver domain too I
>>>> expect that approach should work there too (I think this is your #3?).
>>>
>>> yes
>>>>
>>>>
>>>> Your #2 might be possible, but would probably involve a reasonably
>>>> expensive scan of the various free pools in the hopes of finding the
>>>> block you want, since it isn't designed to be looked up in this way.
>>>>
>>>> I suppose #1 would be something like Linux's CMA allocator -- i.e.
>>>> carving out 1 or more regions on boot and keeping them away from the
>>>> main allocator (but still in e.g. the frametable etc) and a simple way
>>>> to allocate one of the chunks.
>>>
>>> This is interesting. Can I use /memreserve/ logic or something similar to
>>> keeping them away from the main allocator?
>>> But at the stages where we "initialize" domheap we know nothing about
>>> guest domain (domd) and how we need to allocate memory for it (1:1 or
>>> not).
>>
>>
>> The first approach we had for DOM0 1:1 mapping was book-keeping memory at
>> startup. The main allocator wasn't able to use it.
>>
>> But this turn out to be very hackish and not scalable (multiple memory
>> banks...).
>>
>>>>
>>>> So I think any of the approaches you suggest could work, I'd probably
>>>> say #3, #1, #2 in decreasing order of preference.
>>>
>>> I got it.
>>
>>
>> If I correctly understand the solution #3, it may end up to lots of small
>> banks after domains has been created/destroyed multiple
> I do not quite understand why this can happen. Anyway, I need to check
> this assumption.
> I will let you know about result.

Let me describe in detail about solution #3 before answer to your
question. Maybe I missed something in
the first mail. Also the Ian's answer clarified to me some points.
We don't have complete solution for now. We only have temporally
solution in which
we relied on assumptions which might or not be acceptable, but it
seems that the approach in general could work.
If it is true the our target is to rewrite/rework this stuff, to make
it more cleaner and correct from XEN point of view for platforms which
doesn't have a SMMU
even in the case where this stuff never reaches upstream.

To run driver domain (domd) on OMAP5 platform with 1:1 mapping we did
next preparations:
(here I try to explain about memory allocation only, IRQ handling and
other things are out of scope for the current thread)
1. Since the domd can use 128/256/512 Mb of RAM we modified existing
populate_guest_memory() in xc_dom_arm.c to allow to allocate
128/256/512 Mb memory chunks.
2. Since the default rambase (0x40000000) doesn't suitable for us for
some reasons:
- OMAP5 platform has memory mapped registers, starting from 0x4800000
- We have 2 guest domains (domd and domU) so it should be different
rambases for them
we added ability for toolstack to pass it via domain config file.
3. Since for domd we need one contiguous chunk of memory we created a
new function allocate_domd_memory_11() in common/memory.c for the next
purposes:
- To allocate one contiguous memory chunk with specified order (as it
was done for dom0)
- To add this allocated chunk to the guest - domd (via
guest_physmap_add_page(), taking into account that mfn=gpfn)
4. Since we need to allocate memory before any operation with it we
created hook for XENMEM_populate_physmap command. Here we relied on
the assumption
that the domain_id for domd is always 1 (it is not true after the domd
has been created again after destroying).
During first XENMEM_populate_physmap command we set is_privileged=true
and call allocate_domd_memory_11(),
during next commands - call populate_physmap() as a usual. The
is_domain_direct_mapped condition is a our case in populate_physmap().
I know that it is a very, very hacking solution). But, let's continue...

How it works at the moment?
1. Create domd with default rambase_pfn (0x80000000).
2. See what the mfn we got in allocate_domd_memory_11().
3. Set rambase_pfn=mfn in config file.
If the system configuration (N domains, domains memory, etc.) is not
changed, we will always get the same mfn. If we decided to change
something, for example, domd memory we need to repeat steps 2 and 3.
Yes, looks not good.

How it should works?
The approach is to tailor the domd address map to the contiguous
region the allocator gives us. So, a guest rambase (gpfn) must based
on
mfn of a page we have allocated successfully. Without any manual actions.

I think it must be done in next steps:
1. Add a separate command, XENMEM_alloc_memory_11 or something similar
and it should be issued before call xc_dom_rambase_init() in
libxl_dom.c in case of presence
domd_memory_11 property in config file only. This should remove
terrible hook and anything related to d->domain_id=1 in
common/memory.c.
2. Pass returned by XENMEM_alloc_memory_11 result to the xc_dom_rambase_init().

What are the advantages in compare with solution #1 and solution #2.
1. There is no need to add standalone allocator or modifying existing.

Let's return to your question about created/destroyed domd multiple times.
I have tried to do this with solution we have at the moment. I added
some modifications to allow me to destroy/create domd multiple times.
And I have seen that the allocator always returns the same page. This
means that the all memory allocated for domd have been returned to the
heap. Am I right?
Or perhaps did you mean that this may happens with the completed solution?

>
>>
>>>>
>>>> Now, if you were asking for ideas on how to make this stuff
>>>> upstreamable, well that's a harder question. I'm not really sure :-/
>>>
>>> Ideally, I would like to make this stuff upstreamable.
>>
>>
>> As you said on your first mail, this is required to allow DMA on device
>> passthrough when the platform doesn't have an SMMU.
>>
>> I totally understand a such use case for specific embedded product, because
>> you trust your driver domains.
>>
>> On upstream, the main use case of driver domain is too protected the
>> platform from buggy drivers. If the drivers crash, then you can restart only
>> this domain.
>>
>> With a device protected by an SMMU, any DMA requests are safe. Without it,
>> the driver can pretty much do whatever it wants. This could lead to the
>> hypervisor corruption/crash.
>>
>> If a such feature as to come to Xen upstream, this should be hidden to the
>> main users (maybe via compilation option) or require a specific option to
>> use it.
> Completely agree
>
>>
>> Regards.
>>
>> --
>> Julien Grall
>
>
>
> --
>
> Oleksandr Tyshchenko | Embedded Dev
> GlobalLogic
> www.globallogic.com



-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

Re: question about memory allocation for driver domain

[Julien Grall]

On 07/02/2015 01:15, Oleksandr Tyshchenko wrote:
> Hi Julien

Hi Oleksandr,

> On Thu, Feb 5, 2015 at 6:36 PM, Oleksandr Tyshchenko
> Let me describe in detail about solution #3 before answer to your
> question. Maybe I missed something in
> the first mail. Also the Ian's answer clarified to me some points.
> We don't have complete solution for now. We only have temporally
> solution in which
> we relied on assumptions which might or not be acceptable, but it
> seems that the approach in general could work.
> If it is true the our target is to rewrite/rework this stuff, to make
> it more cleaner and correct from XEN point of view for platforms which
> doesn't have a SMMU
> even in the case where this stuff never reaches upstream.
>
> To run driver domain (domd) on OMAP5 platform with 1:1 mapping we did
> next preparations:
> (here I try to explain about memory allocation only, IRQ handling and
> other things are out of scope for the current thread)
> 1. Since the domd can use 128/256/512 Mb of RAM we modified existing
> populate_guest_memory() in xc_dom_arm.c to allow to allocate
> 128/256/512 Mb memory chunks.
> 2. Since the default rambase (0x40000000) doesn't suitable for us for
> some reasons:
> - OMAP5 platform has memory mapped registers, starting from 0x4800000
> - We have 2 guest domains (domd and domU) so it should be different
> rambases for them
> we added ability for toolstack to pass it via domain config file.

While the overlapping is true on this Xen 4.5, we may decide to 
re-arrange the memory layout and put the GIC MMIOs on 0x4800000.

A more generic solution would be to re-use the memory layout of the 
host. So a 1:1 mapping for MMIO and RAM would avoid overlapping with 
possible "virtual" region.

I remembered to talk about it with Ian few months ago but we didn't have 
a practical use case at this time. FWIW, x86 has a similar solution via 
the e820_host param.

> 3. Since for domd we need one contiguous chunk of memory we created a
> new function allocate_domd_memory_11() in common/memory.c for the next
> purposes:
> - To allocate one contiguous memory chunk with specified order (as it
> was done for dom0)
> - To add this allocated chunk to the guest - domd (via
> guest_physmap_add_page(), taking into account that mfn=gpfn)
> 4. Since we need to allocate memory before any operation with it we
> created hook for XENMEM_populate_physmap command. Here we relied on
> the assumption
> that the domain_id for domd is always 1 (it is not true after the domd
> has been created again after destroying).
> During first XENMEM_populate_physmap command we set is_privileged=true
> and call allocate_domd_memory_11(),
> during next commands - call populate_physmap() as a usual. The
> is_domain_direct_mapped condition is a our case in populate_physmap().
> I know that it is a very, very hacking solution). But, let's continue...
>
> How it works at the moment?
> 1. Create domd with default rambase_pfn (0x80000000).
> 2. See what the mfn we got in allocate_domd_memory_11().
> 3. Set rambase_pfn=mfn in config file.
> If the system configuration (N domains, domains memory, etc.) is not
> changed, we will always get the same mfn. If we decided to change
> something, for example, domd memory we need to repeat steps 2 and 3.
> Yes, looks not good.
>
> How it should works?
> The approach is to tailor the domd address map to the contiguous
> region the allocator gives us. So, a guest rambase (gpfn) must based
> on
> mfn of a page we have allocated successfully. Without any manual actions.
>
> I think it must be done in next steps:
> 1. Add a separate command, XENMEM_alloc_memory_11 or something similar
> and it should be issued before call xc_dom_rambase_init() in
> libxl_dom.c in case of presence
> domd_memory_11 property in config file only. This should remove
> terrible hook and anything related to d->domain_id=1 in
> common/memory.c.
> 2. Pass returned by XENMEM_alloc_memory_11 result to the xc_dom_rambase_init().
>
> What are the advantages in compare with solution #1 and solution #2.
> 1. There is no need to add standalone allocator or modifying existing.
>
> Let's return to your question about created/destroyed domd multiple times.
> I have tried to do this with solution we have at the moment. I added
> some modifications to allow me to destroy/create domd multiple times.
> And I have seen that the allocator always returns the same page. This
> means that the all memory allocated for domd have been returned to the
> heap. Am I right?
> Or perhaps did you mean that this may happens with the completed solution?

It seems logical to me that destroy/create domd in a row working fine. 
But this use-case is too simple :).

Let's imagine we decide to start classical domains (i.e no 1:1 mapping) 
before creating domd (the 1:1 domain). As the free memory may be 
sparsed, allocating one large RAM region may not work and therefore the 
domain allocation fail.

On a similar idea, the host RAM may be split on multiple non-contiguous 
banks. In this case, the RAM size of the 1:1 domain cannot be bigger 
than the size of the bank. You will never know which bank is used, as 
IIRC, the allocator behavior change between debug and non-debug build.
We had the same issue on DOM0 before the support of multiple banks has 
been added. It sounds like you may want multiple bank support for an 
upstream use case.

The next problem is ballooning. When the guest balloon out memory, the 
page will be freed by Xen and can be re-used by another domain.

When guest balloon in, Xen will allocate a page (randomly) and therefore 
the mapping won't be anymore IPA (guest physical address) == PA 
(physical address). Any DMA request using this address will read/write 
data from wrong memory.

The last problem but not the least is, depending on which backend you 
are running in the 1:1 domain (such blkback), grant won't be mapped 1:1 
to the guest, so you will have to use swiotlb in order to use the right 
DMA address. For instance, without swiotlb, guest won't be able to use a 
disk partition via blkfront. This because the backend is giving directly 
the grant address to the block driver. To solve this, we have to use 
swiotlb and set specific DMA callback. For now, there are only used for 
DOM0.

I think I've covered all the possible things you have to take care with 
1:1 mapping. Let me know if you need more information on some of them.

Regards,

-- 
Julien Grall

Re: question about memory allocation for driver domain

[Ian Campbell]

On Mon, 2015-02-09 at 16:31 +0800, Julien Grall wrote:
> It seems logical to me that destroy/create domd in a row working fine. 
> But this use-case is too simple :).
> 
> Let's imagine we decide to start classical domains (i.e no 1:1 mapping) 
> before creating domd (the 1:1 domain). As the free memory may be 
> sparsed, allocating one large RAM region may not work and therefore the 
> domain allocation fail.
> 
> On a similar idea, the host RAM may be split on multiple non-contiguous 
> banks. In this case, the RAM size of the 1:1 domain cannot be bigger 
> than the size of the bank. You will never know which bank is used, as 
> IIRC, the allocator behavior change between debug and non-debug build.
> We had the same issue on DOM0 before the support of multiple banks has 
> been added. It sounds like you may want multiple bank support for an 
> upstream use case.

It seems to me that any use of 1:1 memory for !dom0 needs to be from a
preallocated region which is allocated for this purpose at boot and then
reserved for this specific allocation.

e.g. lets imagine a hypervisor option mem_11_reserve=256M,256M,128M
which would, at boot time, allocate 2x 256M contiguous regions and
1x128M one. When building a guest some mechanism (new hypercall, some
other other trickery etc) indicates that the guest being built is
supposed to use one of these regions instead of the usual domheap
allocator.

This would allow for a boot time configurable number of 1:1 regions. I
think this would work for the embedded use case since the domains which
have these special properties are well defined in size and number and so
can be allocated up front.

> The next problem is ballooning. When the guest balloon out memory, the 
> page will be freed by Xen and can be re-used by another domain.

I think we need to do as we do for 1:1 dom0 here and not hand back the
memory on decrease reservation, but instead punch a hole in the p2m but
keep the mfn in reserve.

IOW ballooning is not supported for such domains (we only go as far as
punching the hole to allow for the other usecase of ballooning which is
to make a p2m hole for the Xen backend driver to use for grant maps)

> The last problem but not the least is, depending on which backend you 
> are running in the 1:1 domain (such blkback), grant won't be mapped 1:1 
> to the guest, so you will have to use swiotlb in order to use the right 
> DMA address. For instance, without swiotlb, guest won't be able to use a 
> disk partition via blkfront. This because the backend is giving directly 
> the grant address to the block driver. To solve this, we have to use 
> swiotlb and set specific DMA callback. For now, there are only used for 
> DOM0.

Not much we can do here except extend the dom0 code here to
conditionally enable itself for other domains.

Ian.

Re: question about memory allocation for driver domain

[Julien Grall]

On 09/02/2015 18:53, Ian Campbell wrote:
> On Mon, 2015-02-09 at 16:31 +0800, Julien Grall wrote:
>> It seems logical to me that destroy/create domd in a row working fine.
>> But this use-case is too simple :).
>>
>> Let's imagine we decide to start classical domains (i.e no 1:1 mapping)
>> before creating domd (the 1:1 domain). As the free memory may be
>> sparsed, allocating one large RAM region may not work and therefore the
>> domain allocation fail.
>>
>> On a similar idea, the host RAM may be split on multiple non-contiguous
>> banks. In this case, the RAM size of the 1:1 domain cannot be bigger
>> than the size of the bank. You will never know which bank is used, as
>> IIRC, the allocator behavior change between debug and non-debug build.
>> We had the same issue on DOM0 before the support of multiple banks has
>> been added. It sounds like you may want multiple bank support for an
>> upstream use case.
>
> It seems to me that any use of 1:1 memory for !dom0 needs to be from a
> preallocated region which is allocated for this purpose at boot and then
> reserved for this specific allocation.
>
> e.g. lets imagine a hypervisor option mem_11_reserve=256M,256M,128M
> which would, at boot time, allocate 2x 256M contiguous regions and
> 1x128M one. When building a guest some mechanism (new hypercall, some
> other other trickery etc) indicates that the guest being built is
> supposed to use one of these regions instead of the usual domheap
> allocator.
>
> This would allow for a boot time configurable number of 1:1 regions. I
> think this would work for the embedded use case since the domains which
> have these special properties are well defined in size and number and so
> can be allocated up front.

That seems a fair trade to use 1:1 mapping for domain. And it doesn't 
modify the allocator.


>> The next problem is ballooning. When the guest balloon out memory, the
>> page will be freed by Xen and can be re-used by another domain.
>
> I think we need to do as we do for 1:1 dom0 here and not hand back the
> memory on decrease reservation, but instead punch a hole in the p2m but
> keep the mfn in reserve.

It sounds a fair trade in order to support 1:1 domain mapping.

> IOW ballooning is not supported for such domains (we only go as far as
> punching the hole to allow for the other usecase of ballooning which is
> to make a p2m hole for the Xen backend driver to use for grant maps)

If I'm not mistaken, netback is balloon out to allocate some pages. But 
yeah, as you said, extending the DOM0 1:1 concept would avoid a such 
problem.

The code for the 1:1 mapping in Xen (aside the allocation) is 
domain-agnostic. Oleksandr, I think modifying is_domain_direct_mapped 
(include/asm-arm/domain.h) should be enough here.

>> The last problem but not the least is, depending on which backend you
>> are running in the 1:1 domain (such blkback), grant won't be mapped 1:1
>> to the guest, so you will have to use swiotlb in order to use the right
>> DMA address. For instance, without swiotlb, guest won't be able to use a
>> disk partition via blkfront. This because the backend is giving directly
>> the grant address to the block driver. To solve this, we have to use
>> swiotlb and set specific DMA callback. For now, there are only used for
>> DOM0.
>
> Not much we can do here except extend the dom0 code here to
> conditionally enable itself for other domains.

You mean in the guest kernel? Maybe we have to introduce a new feature 
flags indicating is the domain is using 1:1 mapping or not?

It would help in Xen code too.

Regards,

-- 
Julien Grall

Re: question about memory allocation for driver domain

[Oleksandr Tyshchenko]

On Mon, Feb 9, 2015 at 1:16 PM, Julien Grall <julien.grall@linaro.org> wrote:
>
>
> On 09/02/2015 18:53, Ian Campbell wrote:
>>
>> On Mon, 2015-02-09 at 16:31 +0800, Julien Grall wrote:
>>>
>>> It seems logical to me that destroy/create domd in a row working fine.
>>> But this use-case is too simple :).
>>>
>>> Let's imagine we decide to start classical domains (i.e no 1:1 mapping)
>>> before creating domd (the 1:1 domain). As the free memory may be
>>> sparsed, allocating one large RAM region may not work and therefore the
>>> domain allocation fail.
>>>
>>> On a similar idea, the host RAM may be split on multiple non-contiguous
>>> banks. In this case, the RAM size of the 1:1 domain cannot be bigger
>>> than the size of the bank. You will never know which bank is used, as
>>> IIRC, the allocator behavior change between debug and non-debug build.
>>> We had the same issue on DOM0 before the support of multiple banks has
>>> been added. It sounds like you may want multiple bank support for an
>>> upstream use case.
>>
>>
>> It seems to me that any use of 1:1 memory for !dom0 needs to be from a
>> preallocated region which is allocated for this purpose at boot and then
>> reserved for this specific allocation.
>>
>> e.g. lets imagine a hypervisor option mem_11_reserve=256M,256M,128M
>> which would, at boot time, allocate 2x 256M contiguous regions and
>> 1x128M one. When building a guest some mechanism (new hypercall, some
>> other other trickery etc) indicates that the guest being built is
>> supposed to use one of these regions instead of the usual domheap
>> allocator.
>>
>> This would allow for a boot time configurable number of 1:1 regions. I
>> think this would work for the embedded use case since the domains which
>> have these special properties are well defined in size and number and so
>> can be allocated up front.
>
>
> That seems a fair trade to use 1:1 mapping for domain. And it doesn't modify
> the allocator.
Just to clarify.
The mem_11_reserve option suggested above - can I configure it from
compile command line?
Can I use /memreserve/ logic to not pass these regions to domheap?

>
>
>>> The next problem is ballooning. When the guest balloon out memory, the
>>> page will be freed by Xen and can be re-used by another domain.
>>
>>
>> I think we need to do as we do for 1:1 dom0 here and not hand back the
>> memory on decrease reservation, but instead punch a hole in the p2m but
>> keep the mfn in reserve.
>
>
> It sounds a fair trade in order to support 1:1 domain mapping.
>
>> IOW ballooning is not supported for such domains (we only go as far as
>> punching the hole to allow for the other usecase of ballooning which is
>> to make a p2m hole for the Xen backend driver to use for grant maps)
>
>
> If I'm not mistaken, netback is balloon out to allocate some pages. But
> yeah, as you said, extending the DOM0 1:1 concept would avoid a such
> problem.
>
> The code for the 1:1 mapping in Xen (aside the allocation) is
> domain-agnostic. Oleksandr, I think modifying is_domain_direct_mapped
> (include/asm-arm/domain.h) should be enough here.
Yes. I see that we don't allocate any new pages in populate_physmap
if is_domain_direct_mapped is set. The guest will not get a random page.
So, the ballon - is not a our problem)

>
>>> The last problem but not the least is, depending on which backend you
>>> are running in the 1:1 domain (such blkback), grant won't be mapped 1:1
>>> to the guest, so you will have to use swiotlb in order to use the right
>>> DMA address. For instance, without swiotlb, guest won't be able to use a
>>> disk partition via blkfront. This because the backend is giving directly
>>> the grant address to the block driver. To solve this, we have to use
>>> swiotlb and set specific DMA callback. For now, there are only used for
>>> DOM0.
>>
>>
>> Not much we can do here except extend the dom0 code here to
>> conditionally enable itself for other domains.
>
>
> You mean in the guest kernel? Maybe we have to introduce a new feature flags
> indicating is the domain is using 1:1 mapping or not?
>
> It would help in Xen code too.
>
> Regards,
>
> --
> Julien Grall



-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

Re: question about memory allocation for driver domain

[Ian Campbell]

On Mon, 2015-02-09 at 19:16 +0800, Julien Grall wrote:
> >> The last problem but not the least is, depending on which backend you
> >> are running in the 1:1 domain (such blkback), grant won't be mapped 1:1
> >> to the guest, so you will have to use swiotlb in order to use the right
> >> DMA address. For instance, without swiotlb, guest won't be able to use a
> >> disk partition via blkfront. This because the backend is giving directly
> >> the grant address to the block driver. To solve this, we have to use
> >> swiotlb and set specific DMA callback. For now, there are only used for
> >> DOM0.
> >
> > Not much we can do here except extend the dom0 code here to
> > conditionally enable itself for other domains.
> 
> You mean in the guest kernel? Maybe we have to introduce a new feature 
> flags indicating is the domain is using 1:1 mapping or not?

Something along those lines I would expect.

I'm not sure it fits, but this has some similarities with the DTB
bindings used to indicate to the guest that a device is protected (in
the SMMU sense) which you did a while back. It's possible we could reuse
or extend that interface -- after all 1:1 is somewhat like having an
SMMU, at least to the extent that the things the guest is expected to do
are the same.

Ian.

Re: question about memory allocation for driver domain

[Julien Grall]

Hi Ian,

On 09/02/2015 21:08, Ian Campbell wrote:
> On Mon, 2015-02-09 at 19:16 +0800, Julien Grall wrote:
>>>> The last problem but not the least is, depending on which backend you
>>>> are running in the 1:1 domain (such blkback), grant won't be mapped 1:1
>>>> to the guest, so you will have to use swiotlb in order to use the right
>>>> DMA address. For instance, without swiotlb, guest won't be able to use a
>>>> disk partition via blkfront. This because the backend is giving directly
>>>> the grant address to the block driver. To solve this, we have to use
>>>> swiotlb and set specific DMA callback. For now, there are only used for
>>>> DOM0.
>>>
>>> Not much we can do here except extend the dom0 code here to
>>> conditionally enable itself for other domains.
>>
>> You mean in the guest kernel? Maybe we have to introduce a new feature
>> flags indicating is the domain is using 1:1 mapping or not?
>
> Something along those lines I would expect.
>
> I'm not sure it fits, but this has some similarities with the DTB
> bindings used to indicate to the guest that a device is protected (in
> the SMMU sense) which you did a while back. It's possible we could reuse
> or extend that interface -- after all 1:1 is somewhat like having an
> SMMU, at least to the extent that the things the guest is expected to do
> are the same.

1:1 domain and having SMMU is not the same, the former requires swiotlb 
while the latter not.

When the device is protected, we can avoid to use the Xen DMA ops (based 
on swiotlb). For this purpose, the patch was added a property in the 
device tree node of device protected by an SMMU.

In the case of the 1:1 mapping we definitely want to use swiotlb, for 
devices using DMA in 1:1 domain. This will be for the whole guest, so 
adding a property per device seems pointless.

Reusing the same property would mean that the common use case (device 
passthrough protected by an SMMU) will require an little extra work. It 
doesn't sounds right to me.

After all, a per-domain seems the best things to do. It would also helps 
on platform that doesn't require 1:1 mapping for DOM0 (I'm think about 
Cavium, and maybe Seattle?). This would increase performance in general.

Regards,

-- 
Julien Grall

Re: question about memory allocation for driver domain

[Oleksandr Tyshchenko]

Hi Ian

On Mon, Feb 9, 2015 at 12:53 PM, Ian Campbell <ian.campbell@citrix.com> wrote:
> On Mon, 2015-02-09 at 16:31 +0800, Julien Grall wrote:
>> It seems logical to me that destroy/create domd in a row working fine.
>> But this use-case is too simple :).
>>
>> Let's imagine we decide to start classical domains (i.e no 1:1 mapping)
>> before creating domd (the 1:1 domain). As the free memory may be
>> sparsed, allocating one large RAM region may not work and therefore the
>> domain allocation fail.
>>
>> On a similar idea, the host RAM may be split on multiple non-contiguous
>> banks. In this case, the RAM size of the 1:1 domain cannot be bigger
>> than the size of the bank. You will never know which bank is used, as
>> IIRC, the allocator behavior change between debug and non-debug build.
>> We had the same issue on DOM0 before the support of multiple banks has
>> been added. It sounds like you may want multiple bank support for an
>> upstream use case.
>
> It seems to me that any use of 1:1 memory for !dom0 needs to be from a
> preallocated region which is allocated for this purpose at boot and then
> reserved for this specific allocation.
>
> e.g. lets imagine a hypervisor option mem_11_reserve=256M,256M,128M
> which would, at boot time, allocate 2x 256M contiguous regions and
> 1x128M one. When building a guest some mechanism (new hypercall, some
> other other trickery etc) indicates that the guest being built is
> supposed to use one of these regions instead of the usual domheap
> allocator.
>
> This would allow for a boot time configurable number of 1:1 regions. I
> think this would work for the embedded use case since the domains which
> have these special properties are well defined in size and number and so
> can be allocated up front.
Sounds reasonable.

>
>> The next problem is ballooning. When the guest balloon out memory, the
>> page will be freed by Xen and can be re-used by another domain.
>
> I think we need to do as we do for 1:1 dom0 here and not hand back the
> memory on decrease reservation, but instead punch a hole in the p2m but
> keep the mfn in reserve.
>
> IOW ballooning is not supported for such domains (we only go as far as
> punching the hole to allow for the other usecase of ballooning which is
> to make a p2m hole for the Xen backend driver to use for grant maps)
>
>> The last problem but not the least is, depending on which backend you
>> are running in the 1:1 domain (such blkback), grant won't be mapped 1:1
>> to the guest, so you will have to use swiotlb in order to use the right
>> DMA address. For instance, without swiotlb, guest won't be able to use a
>> disk partition via blkfront. This because the backend is giving directly
>> the grant address to the block driver. To solve this, we have to use
>> swiotlb and set specific DMA callback. For now, there are only used for
>> DOM0.
>
> Not much we can do here except extend the dom0 code here to
> conditionally enable itself for other domains.
Agree. There are no problem with swiotlb and xen_dma_ops. We already
have working backends in domd: fb, vkbd, sound.

>
> Ian.
>



-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

Re: question about memory allocation for driver domain

[Oleksandr Tyshchenko]

On Mon, Feb 9, 2015 at 10:31 AM, Julien Grall <julien.grall@linaro.org> wrote:
>
>
> On 07/02/2015 01:15, Oleksandr Tyshchenko wrote:
>>
>> Hi Julien
>
>
> Hi Oleksandr,
Hi Julien

>
>> On Thu, Feb 5, 2015 at 6:36 PM, Oleksandr Tyshchenko
>> Let me describe in detail about solution #3 before answer to your
>> question. Maybe I missed something in
>> the first mail. Also the Ian's answer clarified to me some points.
>> We don't have complete solution for now. We only have temporally
>> solution in which
>> we relied on assumptions which might or not be acceptable, but it
>> seems that the approach in general could work.
>> If it is true the our target is to rewrite/rework this stuff, to make
>> it more cleaner and correct from XEN point of view for platforms which
>> doesn't have a SMMU
>> even in the case where this stuff never reaches upstream.
>>
>> To run driver domain (domd) on OMAP5 platform with 1:1 mapping we did
>> next preparations:
>> (here I try to explain about memory allocation only, IRQ handling and
>> other things are out of scope for the current thread)
>> 1. Since the domd can use 128/256/512 Mb of RAM we modified existing
>> populate_guest_memory() in xc_dom_arm.c to allow to allocate
>> 128/256/512 Mb memory chunks.
>> 2. Since the default rambase (0x40000000) doesn't suitable for us for
>> some reasons:
>> - OMAP5 platform has memory mapped registers, starting from 0x4800000
>> - We have 2 guest domains (domd and domU) so it should be different
>> rambases for them
>> we added ability for toolstack to pass it via domain config file.
>
>
> While the overlapping is true on this Xen 4.5, we may decide to re-arrange
> the memory layout and put the GIC MMIOs on 0x4800000.
>
> A more generic solution would be to re-use the memory layout of the host. So
> a 1:1 mapping for MMIO and RAM would avoid overlapping with possible
> "virtual" region.
>
> I remembered to talk about it with Ian few months ago but we didn't have a
> practical use case at this time. FWIW, x86 has a similar solution via the
> e820_host param.
>
>
>> 3. Since for domd we need one contiguous chunk of memory we created a
>> new function allocate_domd_memory_11() in common/memory.c for the next
>> purposes:
>> - To allocate one contiguous memory chunk with specified order (as it
>> was done for dom0)
>> - To add this allocated chunk to the guest - domd (via
>> guest_physmap_add_page(), taking into account that mfn=gpfn)
>> 4. Since we need to allocate memory before any operation with it we
>> created hook for XENMEM_populate_physmap command. Here we relied on
>> the assumption
>> that the domain_id for domd is always 1 (it is not true after the domd
>> has been created again after destroying).
>> During first XENMEM_populate_physmap command we set is_privileged=true
>> and call allocate_domd_memory_11(),
>> during next commands - call populate_physmap() as a usual. The
>> is_domain_direct_mapped condition is a our case in populate_physmap().
>> I know that it is a very, very hacking solution). But, let's continue...
>>
>> How it works at the moment?
>> 1. Create domd with default rambase_pfn (0x80000000).
>> 2. See what the mfn we got in allocate_domd_memory_11().
>> 3. Set rambase_pfn=mfn in config file.
>> If the system configuration (N domains, domains memory, etc.) is not
>> changed, we will always get the same mfn. If we decided to change
>> something, for example, domd memory we need to repeat steps 2 and 3.
>> Yes, looks not good.
>>
>> How it should works?
>> The approach is to tailor the domd address map to the contiguous
>> region the allocator gives us. So, a guest rambase (gpfn) must based
>> on
>> mfn of a page we have allocated successfully. Without any manual actions.
>>
>> I think it must be done in next steps:
>> 1. Add a separate command, XENMEM_alloc_memory_11 or something similar
>> and it should be issued before call xc_dom_rambase_init() in
>> libxl_dom.c in case of presence
>> domd_memory_11 property in config file only. This should remove
>> terrible hook and anything related to d->domain_id=1 in
>> common/memory.c.
>> 2. Pass returned by XENMEM_alloc_memory_11 result to the
>> xc_dom_rambase_init().
>>
>> What are the advantages in compare with solution #1 and solution #2.
>> 1. There is no need to add standalone allocator or modifying existing.
>>
>> Let's return to your question about created/destroyed domd multiple times.
>> I have tried to do this with solution we have at the moment. I added
>> some modifications to allow me to destroy/create domd multiple times.
>> And I have seen that the allocator always returns the same page. This
>> means that the all memory allocated for domd have been returned to the
>> heap. Am I right?
>> Or perhaps did you mean that this may happens with the completed solution?
>
>
> It seems logical to me that destroy/create domd in a row working fine. But
> this use-case is too simple :).
>
> Let's imagine we decide to start classical domains (i.e no 1:1 mapping)
> before creating domd (the 1:1 domain). As the free memory may be sparsed,
> allocating one large RAM region may not work and therefore the domain
> allocation fail.
>
> On a similar idea, the host RAM may be split on multiple non-contiguous
> banks. In this case, the RAM size of the 1:1 domain cannot be bigger than
> the size of the bank. You will never know which bank is used, as IIRC, the
> allocator behavior change between debug and non-debug build.
> We had the same issue on DOM0 before the support of multiple banks has been
> added. It sounds like you may want multiple bank support for an upstream use
> case.
I understand the possible issue. I need to think about it.

>
> The next problem is ballooning. When the guest balloon out memory, the page
> will be freed by Xen and can be re-used by another domain.
>
> When guest balloon in, Xen will allocate a page (randomly) and therefore the
> mapping won't be anymore IPA (guest physical address) == PA (physical
> address). Any DMA request using this address will read/write data from wrong
> memory.
Is it possible to disable ballon driver for domd?

>
> The last problem but not the least is, depending on which backend you are
> running in the 1:1 domain (such blkback), grant won't be mapped 1:1 to the
> guest, so you will have to use swiotlb in order to use the right DMA
> address. For instance, without swiotlb, guest won't be able to use a disk
> partition via blkfront. This because the backend is giving directly the
> grant address to the block driver. To solve this, we have to use swiotlb and
> set specific DMA callback. For now, there are only used for DOM0.
We are using SWIOTLB for domd too.

>
> I think I've covered all the possible things you have to take care with 1:1
> mapping. Let me know if you need more information on some of them.

OK. Thank you.



>
> Regards,
>
> --
> Julien Grall



-- 

Oleksandr Tyshchenko | Embedded Dev
GlobalLogic
www.globallogic.com

Re: question about memory allocation for driver domain

[Julien Grall]

On 09/02/2015 20:02, Oleksandr Tyshchenko wrote:
> On Mon, Feb 9, 2015 at 10:31 AM, Julien Grall <julien.grall@linaro.org> wrote:
>> On 07/02/2015 01:15, Oleksandr Tyshchenko wrote:
> Hi Julien

Hi Oleksandr,

>>> On Thu, Feb 5, 2015 at 6:36 PM, Oleksandr Tyshchenko
>>
>> The next problem is ballooning. When the guest balloon out memory, the page
>> will be freed by Xen and can be re-used by another domain.
>>
>> When guest balloon in, Xen will allocate a page (randomly) and therefore the
>> mapping won't be anymore IPA (guest physical address) == PA (physical
>> address). Any DMA request using this address will read/write data from wrong
>> memory.
> Is it possible to disable ballon driver for domd?

I think updating is_domain_direct_mapped is enough here.

We won't support memory hotplug for this kind of domain.

>>
>> The last problem but not the least is, depending on which backend you are
>> running in the 1:1 domain (such blkback), grant won't be mapped 1:1 to the
>> guest, so you will have to use swiotlb in order to use the right DMA
>> address. For instance, without swiotlb, guest won't be able to use a disk
>> partition via blkfront. This because the backend is giving directly the
>> grant address to the block driver. To solve this, we have to use swiotlb and
>> set specific DMA callback. For now, there are only used for DOM0.
> We are using SWIOTLB for domd too.

Good :).


Regards,

-- 
Julien Grall