[RFC 0/2] xen: Clarify the page granularity for the hypercall

[RFC 0/2] xen: Clarify the page granularity for the hypercall

[Julien Grall]

Hi all,

ARM64 is able to support both 64KB and 4KB page granularity. With the upcoming
support of Linux guest with 64KB page granularity, the in-tree documentation
needs to be clarify in order to avoid mixing granularity.

I'm not sure if the wording is clear and correct, hence, the RFC.

Regards,

Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Tim Deegan <tim@xen.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>

Julien Grall (1):
  xen/public: grant-table: Specificy the size of the grant

Stefano Stabellini (1):
  xen/mm: Clarify the granularity for each Frame Number

 xen/include/public/grant_table.h |  5 +++++
 xen/include/xen/mm.h             | 20 ++++++++++++++------
 2 files changed, 19 insertions(+), 6 deletions(-)

-- 
2.1.4

[RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Julien Grall]

From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>

ARM64 is able to support 64KB and 4KB page granularities. While the guest
will support both granularities, Xen and hypercall interface will always
be in 4KB.

Signed-off-by: Stefano Stabellini <stefano.stabellini@citrix.com>
Signed-off-by: Julien Grall <julien.grall@citrix.com>

---
Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Tim Deegan <tim@xen.org>
Cc: Andrew Cooper <andrew.cooper3@citrix.com>

I'm missing one term for Linux Frame Number but always in 4KB
granularity. It's necessary in few places such as the balloon code where
we need to map a Linux 4KB Frame Number into a Machine Frame Number.

I was thinking to name it xen_pfn.
---
 xen/include/xen/mm.h | 20 ++++++++++++++------
 1 file changed, 14 insertions(+), 6 deletions(-)

diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
index 876d370..8dfd61a 100644
--- a/xen/include/xen/mm.h
+++ b/xen/include/xen/mm.h
@@ -15,16 +15,24 @@
  *
  * mfn: Machine Frame Number
  *   The values Xen puts into its own pagetables.  This is the host physical
- *   memory address space with RAM, MMIO etc.
+ *   memory address space with RAM, MMIO etc. Always 4KB granularity.
  *
  * gfn: Guest Frame Number
- *   The values a guest puts in its own pagetables.  For an auto-translated
- *   guest (hardware assisted with 2nd stage translation, or shadowed), gfn !=
- *   mfn.  For a non-translated guest which is aware of Xen, gfn == mfn.
+ *   The values a guest puts in its own pagetables, except for 64KB
+ *   granularity. Gfns are always based on 4KB granularity, while actually
+ *   the guest could use other granularities.  For an auto-translated guest
+ *   (hardware assisted with 2nd stage translation, or shadowed), gfn != mfn.
+ *   For a non-translated guest which is aware of Xen, gfn == mfn.
+ *   Hypercalls take gfns, not mfns, as parameters unless clearly specified
+ *   otherwise.
  *
  * pfn: Pseudophysical Frame Number
- *   A linear idea of a guest physical address space. For an auto-translated
- *   guest, pfn == gfn while for a non-translated guest, pfn != gfn.
+ *   A linear idea of a guest physical address space. Pfns are in guest
+ *   granularity, which can be 64KB or 4KB. PV guests must only use 4KB
+ *   granularity. For an auto-translated guest, pfn == gfn << shift,
+ *   where the shift is the different between the Xen and Linux page
+ *   granularity, while for a non-translated guest, pfn != gfn. Pfns are
+ *   internal to the guest and are not passed to hypercalls.
  *
  * WARNING: Some of these terms have changed over time while others have been
  * used inconsistently, meaning that a lot of existing code does not match the
-- 
2.1.4

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Andrew Cooper]

On 05/08/15 12:28, Julien Grall wrote:
> From: Stefano Stabellini <stefano.stabellini@eu.citrix.com>
>
> ARM64 is able to support 64KB and 4KB page granularities. While the guest
> will support both granularities, Xen and hypercall interface will always
> be in 4KB.
>
> Signed-off-by: Stefano Stabellini <stefano.stabellini@citrix.com>
> Signed-off-by: Julien Grall <julien.grall@citrix.com>
>
> ---
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Keir Fraser <keir@xen.org>
> Cc: Tim Deegan <tim@xen.org>
> Cc: Andrew Cooper <andrew.cooper3@citrix.com>
>
> I'm missing one term for Linux Frame Number but always in 4KB
> granularity. It's necessary in few places such as the balloon code where
> we need to map a Linux 4KB Frame Number into a Machine Frame Number.
>
> I was thinking to name it xen_pfn.
> ---
>  xen/include/xen/mm.h | 20 ++++++++++++++------
>  1 file changed, 14 insertions(+), 6 deletions(-)
>
> diff --git a/xen/include/xen/mm.h b/xen/include/xen/mm.h
> index 876d370..8dfd61a 100644
> --- a/xen/include/xen/mm.h
> +++ b/xen/include/xen/mm.h
> @@ -15,16 +15,24 @@
>   *
>   * mfn: Machine Frame Number
>   *   The values Xen puts into its own pagetables.  This is the host physical
> - *   memory address space with RAM, MMIO etc.
> + *   memory address space with RAM, MMIO etc. Always 4KB granularity.
>   *
>   * gfn: Guest Frame Number
> - *   The values a guest puts in its own pagetables.  For an auto-translated
> - *   guest (hardware assisted with 2nd stage translation, or shadowed), gfn !=
> - *   mfn.  For a non-translated guest which is aware of Xen, gfn == mfn.
> + *   The values a guest puts in its own pagetables, except for 64KB
> + *   granularity. Gfns are always based on 4KB granularity, while actually
> + *   the guest could use other granularities.  For an auto-translated guest
> + *   (hardware assisted with 2nd stage translation, or shadowed), gfn != mfn.
> + *   For a non-translated guest which is aware of Xen, gfn == mfn.
> + *   Hypercalls take gfns, not mfns, as parameters unless clearly specified
> + *   otherwise.
>   *
>   * pfn: Pseudophysical Frame Number
> - *   A linear idea of a guest physical address space. For an auto-translated
> - *   guest, pfn == gfn while for a non-translated guest, pfn != gfn.
> + *   A linear idea of a guest physical address space. Pfns are in guest
> + *   granularity, which can be 64KB or 4KB. PV guests must only use 4KB
> + *   granularity. For an auto-translated guest, pfn == gfn << shift,
> + *   where the shift is the different between the Xen and Linux page
> + *   granularity, while for a non-translated guest, pfn != gfn. Pfns are
> + *   internal to the guest and are not passed to hypercalls.
>   *
>   * WARNING: Some of these terms have changed over time while others have been
>   * used inconsistently, meaning that a lot of existing code does not match the

I am not certain whether this is relevant information in these locations
specifically.  These descriptions are for the address spaces themselves,
rather than for the representations therewithin.

64K granularity is also similar to 2M/1G superpages in their handling,
the difference being that 64K can't be subdivided if necessary?

I think a section about granularity is worthwhile, but probably a
separate paragraph.  I think it is also worth keeping Xen's idea of
memory all at 4K, and in cases where 64K is in use, require appropriate
alignment in the parameter.

~Andrew

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Ian Campbell]

On Wed, 2015-08-05 at 12:40 +0100, Andrew Cooper wrote:
> 
> 64K granularity is also similar to 2M/1G superpages in their handling,
> the difference being that 64K can't be subdivided if necessary?

64K is actually a separate basic "granule" (to use the ARM term), i.e.
alternative to the 4K leaf page size, you can still have e.g. 32M super
pages pages if them etc (assuming there are still 512 PTEs per page, I
didn't actually check).

So I think thinking of 64K as a "superpage" while superficially correct
would probably lead to some misunderstanding or confusion at some point.

FWIW there is also a 16K granule available on ARM systems. PPC supports
256K pages too...


Ian

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Julien Grall]

On 05/08/15 12:40, Andrew Cooper wrote:
> I think a section about granularity is worthwhile, but probably a
> separate paragraph.  I think it is also worth keeping Xen's idea of
> memory all at 4K, and in cases where 64K is in use, require appropriate
> alignment in the parameter.

Which would confuse the reader because PFN which, based on the
description, is the OS Frame Number. This frame will always be in the
granularity of the OS.

So we need to introduce the concept of in each definition. This patch
makes clear that MFN and GFN is always 4KB and PFN may vary.

Regards,

-- 
Julien Grall

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Andrew Cooper]

On 05/08/15 13:36, Julien Grall wrote:
> On 05/08/15 12:40, Andrew Cooper wrote:
>> I think a section about granularity is worthwhile, but probably a
>> separate paragraph.  I think it is also worth keeping Xen's idea of
>> memory all at 4K, and in cases where 64K is in use, require appropriate
>> alignment in the parameter.
> Which would confuse the reader because PFN which, based on the
> description, is the OS Frame Number. This frame will always be in the
> granularity of the OS.

"A linear idea of a guest physical address space." says nothing about
the OS.  It is purely a Xen concept, as described here.

>
> So we need to introduce the concept of in each definition. This patch
> makes clear that MFN and GFN is always 4KB and PFN may vary.

Is (or rather will) a 4K dom0 able to make 4K mappings of a 64K domU? 
How is a 64K dom0 expected to make mappings of a 4K domU?

The primary use of "pfn" in Xen is logdirty tracking (which ARM doesn't
have yet), but will have to be set at the minimum granularity of the
toolstack domain, domU and the logdirty ABI which currently is assumed
to be 4K pages because of its x86 heritage.

~Andrew

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Julien Grall]

Hi,

On 05/08/15 13:46, Andrew Cooper wrote:
> On 05/08/15 13:36, Julien Grall wrote:
>> On 05/08/15 12:40, Andrew Cooper wrote:
>>> I think a section about granularity is worthwhile, but probably a
>>> separate paragraph.  I think it is also worth keeping Xen's idea of
>>> memory all at 4K, and in cases where 64K is in use, require appropriate
>>> alignment in the parameter.
>> Which would confuse the reader because PFN which, based on the
>> description, is the OS Frame Number. This frame will always be in the
>> granularity of the OS.
> 
> "A linear idea of a guest physical address space." says nothing about
> the OS.  It is purely a Xen concept, as described here.

Right. Thank you for your explanation IRL.

I will have to find another place to clear specify the granularity of
the hypercalls.

>>
>> So we need to introduce the concept of in each definition. This patch
>> makes clear that MFN and GFN is always 4KB and PFN may vary.
> 
> Is (or rather will) a 4K dom0 able to make 4K mappings of a 64K domU? 
> How is a 64K dom0 expected to make mappings of a 4K domU?

The Xen interface will stay 4K even with 64K guest. We have to support
64K guest/dom0 on the current Xen because some distro may do the choice
to only ship 64K.

In my current implementation of Linux 64K support (see [1]), there is no
changes in Xen (hypervisor and tools). Linux is breaking the 64K page in
4K chunk.

When the backend is 64K, it will map the foreign 4K at the top of a 64K
page. It's a waste of memory, but it's easier to implement and it's
still and improvement compare to have Linux crashing at boot.

Note that there is lots of room of improvement but I'd like to get a
series as soon as possible. That doesn't mean we should have a hack,
just something sensible and working.

> The primary use of "pfn" in Xen is logdirty tracking (which ARM doesn't
> have yet), but will have to be set at the minimum granularity of the
> toolstack domain, domU and the logdirty ABI which currently is assumed
> to be 4K pages because of its x86 heritage.

Which will be fine for ARM given that ARM32 only supports 4K.

Regards,

[1] https://lkml.org/lkml/2015/7/9/611

-- 
Julien Grall

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Jan Beulich]

>>> On 05.08.15 at 15:18, <julien.grall@citrix.com> wrote:
> On 05/08/15 13:46, Andrew Cooper wrote:
>> On 05/08/15 13:36, Julien Grall wrote:
>>> So we need to introduce the concept of in each definition. This patch
>>> makes clear that MFN and GFN is always 4KB and PFN may vary.
>> 
>> Is (or rather will) a 4K dom0 able to make 4K mappings of a 64K domU? 
>> How is a 64K dom0 expected to make mappings of a 4K domU?
> 
> The Xen interface will stay 4K even with 64K guest. We have to support
> 64K guest/dom0 on the current Xen because some distro may do the choice
> to only ship 64K.

Interesting. Does Linux on ARM not require any atomic page table
entry updates? I ask because I can't see how you would emulate
such when you need to deal with 16 of them at a time.

> In my current implementation of Linux 64K support (see [1]), there is no
> changes in Xen (hypervisor and tools). Linux is breaking the 64K page in
> 4K chunk.
> 
> When the backend is 64K, it will map the foreign 4K at the top of a 64K
> page. It's a waste of memory, but it's easier to implement and it's
> still and improvement compare to have Linux crashing at boot.

Waste of memory? You're only mapping an existing chunk of memory.
DYM waste of address space?

Jan

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Julien Grall]

Hi Jan,

On 12/08/2015 08:16, Jan Beulich wrote:
>>>> On 05.08.15 at 15:18, <julien.grall@citrix.com> wrote:
>> On 05/08/15 13:46, Andrew Cooper wrote:
>>> On 05/08/15 13:36, Julien Grall wrote:
>>>> So we need to introduce the concept of in each definition. This patch
>>>> makes clear that MFN and GFN is always 4KB and PFN may vary.
>>>
>>> Is (or rather will) a 4K dom0 able to make 4K mappings of a 64K domU?
>>> How is a 64K dom0 expected to make mappings of a 4K domU?
>>
>> The Xen interface will stay 4K even with 64K guest. We have to support
>> 64K guest/dom0 on the current Xen because some distro may do the choice
>> to only ship 64K.
>
> Interesting. Does Linux on ARM not require any atomic page table
> entry updates? I ask because I can't see how you would emulate
> such when you need to deal with 16 of them at a time.

I'm not sure to understand this question.

ARM64 is able to support different page granularity (4KB, 16KB and 
64KB). You have to setup the page table registers during the boot in 
order to specify the granularity used for the whole page table.

In Linux, the page size is chosen at build time and therefore not 
possible to switch it automatically.

>
>> In my current implementation of Linux 64K support (see [1]), there is no
>> changes in Xen (hypervisor and tools). Linux is breaking the 64K page in
>> 4K chunk.
>>
>> When the backend is 64K, it will map the foreign 4K at the top of a 64K
>> page. It's a waste of memory, but it's easier to implement and it's
>> still and improvement compare to have Linux crashing at boot.
>
> Waste of memory? You're only mapping an existing chunk of memory.
> DYM waste of address space?

No, I really meant waste of memory. The current grant API in Linux is 
allocating one Linux Page per grant. Although the grant is always 4K, so 
we won't be able to make use of the 60K for anything as long as we use 
this page for a grant.

So if the grant is pre-allocated (such as for PV block), we won't be 
able use nr_grant * 60KB memory.

It's in my plan to improve the memory usage but we wanted something 
working given that today booting Linux with 64KB page granularity is 
crashing on Xen.

Regards,

-- 
Julien Grall

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Jan Beulich]

>>> On 12.08.15 at 11:57, <julien.grall@citrix.com> wrote:
> On 12/08/2015 08:16, Jan Beulich wrote:
>>>>> On 05.08.15 at 15:18, <julien.grall@citrix.com> wrote:
>>> On 05/08/15 13:46, Andrew Cooper wrote:
>>>> On 05/08/15 13:36, Julien Grall wrote:
>>>>> So we need to introduce the concept of in each definition. This patch
>>>>> makes clear that MFN and GFN is always 4KB and PFN may vary.
>>>>
>>>> Is (or rather will) a 4K dom0 able to make 4K mappings of a 64K domU?
>>>> How is a 64K dom0 expected to make mappings of a 4K domU?
>>>
>>> The Xen interface will stay 4K even with 64K guest. We have to support
>>> 64K guest/dom0 on the current Xen because some distro may do the choice
>>> to only ship 64K.
>>
>> Interesting. Does Linux on ARM not require any atomic page table
>> entry updates? I ask because I can't see how you would emulate
>> such when you need to deal with 16 of them at a time.
> 
> I'm not sure to understand this question.
> 
> ARM64 is able to support different page granularity (4KB, 16KB and 
> 64KB). You have to setup the page table registers during the boot in 
> order to specify the granularity used for the whole page table.

But you said you use 4k pages in Xen nevertheless. I.e. page tables
would still be at 4k granularity, i.e. you'd need to update 16 entries
for a single 64k page. Or can you have 64k pages in L1 and 4k pages
in L2?

>>> In my current implementation of Linux 64K support (see [1]), there is no
>>> changes in Xen (hypervisor and tools). Linux is breaking the 64K page in
>>> 4K chunk.
>>>
>>> When the backend is 64K, it will map the foreign 4K at the top of a 64K
>>> page. It's a waste of memory, but it's easier to implement and it's
>>> still and improvement compare to have Linux crashing at boot.
>>
>> Waste of memory? You're only mapping an existing chunk of memory.
>> DYM waste of address space?
> 
> No, I really meant waste of memory. The current grant API in Linux is 
> allocating one Linux Page per grant. Although the grant is always 4K, so 
> we won't be able to make use of the 60K for anything as long as we use 
> this page for a grant.
> 
> So if the grant is pre-allocated (such as for PV block), we won't be 
> able use nr_grant * 60KB memory.

I still don't follow - grant mappings ought to be done into ballooned
(i.e. empty) pages, i.e. no memory would get wasted unless there
are too few balloon pages available.

Jan

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Julien Grall]

On 12/08/15 11:33, Jan Beulich wrote:
>>>> On 12.08.15 at 11:57, <julien.grall@citrix.com> wrote:
>> On 12/08/2015 08:16, Jan Beulich wrote:
>>>>>> On 05.08.15 at 15:18, <julien.grall@citrix.com> wrote:
>>>> On 05/08/15 13:46, Andrew Cooper wrote:
>>>>> On 05/08/15 13:36, Julien Grall wrote:
>>>>>> So we need to introduce the concept of in each definition. This patch
>>>>>> makes clear that MFN and GFN is always 4KB and PFN may vary.
>>>>>
>>>>> Is (or rather will) a 4K dom0 able to make 4K mappings of a 64K domU?
>>>>> How is a 64K dom0 expected to make mappings of a 4K domU?
>>>>
>>>> The Xen interface will stay 4K even with 64K guest. We have to support
>>>> 64K guest/dom0 on the current Xen because some distro may do the choice
>>>> to only ship 64K.
>>>
>>> Interesting. Does Linux on ARM not require any atomic page table
>>> entry updates? I ask because I can't see how you would emulate
>>> such when you need to deal with 16 of them at a time.
>>
>> I'm not sure to understand this question.
>>
>> ARM64 is able to support different page granularity (4KB, 16KB and 
>> 64KB). You have to setup the page table registers during the boot in 
>> order to specify the granularity used for the whole page table.
> 
> But you said you use 4k pages in Xen nevertheless. I.e. page tables
> would still be at 4k granularity, i.e. you'd need to update 16 entries
> for a single 64k page. Or can you have 64k pages in L1 and 4k pages
> in L2?

The page table for each stage are completely dissociated. So you can use
a different page granularity for Xen and Linux.

>>>> In my current implementation of Linux 64K support (see [1]), there is no
>>>> changes in Xen (hypervisor and tools). Linux is breaking the 64K page in
>>>> 4K chunk.
>>>>
>>>> When the backend is 64K, it will map the foreign 4K at the top of a 64K
>>>> page. It's a waste of memory, but it's easier to implement and it's
>>>> still and improvement compare to have Linux crashing at boot.
>>>
>>> Waste of memory? You're only mapping an existing chunk of memory.
>>> DYM waste of address space?
>>
>> No, I really meant waste of memory. The current grant API in Linux is 
>> allocating one Linux Page per grant. Although the grant is always 4K, so 
>> we won't be able to make use of the 60K for anything as long as we use 
>> this page for a grant.
>>
>> So if the grant is pre-allocated (such as for PV block), we won't be 
>> able use nr_grant * 60KB memory.
> 
> I still don't follow - grant mappings ought to be done into ballooned
> (i.e. empty) pages, i.e. no memory would get wasted unless there
> are too few balloon pages available.

Everything balloon out is less memory that can be used by Linux. If we
are only using 1/15 of the balloon out page that a huge waste of memory
for me.

-- 
Julien Grall

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Jan Beulich]

>>> On 12.08.15 at 13:13, <julien.grall@citrix.com> wrote:
> On 12/08/15 11:33, Jan Beulich wrote:
>>>>> On 12.08.15 at 11:57, <julien.grall@citrix.com> wrote:
>>> On 12/08/2015 08:16, Jan Beulich wrote:
>>>>>>> On 05.08.15 at 15:18, <julien.grall@citrix.com> wrote:
>>>>> When the backend is 64K, it will map the foreign 4K at the top of a 64K
>>>>> page. It's a waste of memory, but it's easier to implement and it's
>>>>> still and improvement compare to have Linux crashing at boot.
>>>>
>>>> Waste of memory? You're only mapping an existing chunk of memory.
>>>> DYM waste of address space?
>>>
>>> No, I really meant waste of memory. The current grant API in Linux is 
>>> allocating one Linux Page per grant. Although the grant is always 4K, so 
>>> we won't be able to make use of the 60K for anything as long as we use 
>>> this page for a grant.
>>>
>>> So if the grant is pre-allocated (such as for PV block), we won't be 
>>> able use nr_grant * 60KB memory.
>> 
>> I still don't follow - grant mappings ought to be done into ballooned
>> (i.e. empty) pages, i.e. no memory would get wasted unless there
>> are too few balloon pages available.
> 
> Everything balloon out is less memory that can be used by Linux. If we
> are only using 1/15 of the balloon out page that a huge waste of memory
> for me.

I still don't get it: A ballooned page does not represent any memory
(and at least on x86 iirc we start PV domains with 8Mb sort-of-
pre-ballooned space). Where is the loss you're talking about coming
from? Or, wait - you don't have PoD (yet) on ARM, so you have no
choice.

Jan

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Julien Grall]

On 12/08/15 12:58, Jan Beulich wrote:
>>>> On 12.08.15 at 13:13, <julien.grall@citrix.com> wrote:
>> On 12/08/15 11:33, Jan Beulich wrote:
>>>>>> On 12.08.15 at 11:57, <julien.grall@citrix.com> wrote:
>>>> On 12/08/2015 08:16, Jan Beulich wrote:
>>>>>>>> On 05.08.15 at 15:18, <julien.grall@citrix.com> wrote:
>>>>>> When the backend is 64K, it will map the foreign 4K at the top of a 64K
>>>>>> page. It's a waste of memory, but it's easier to implement and it's
>>>>>> still and improvement compare to have Linux crashing at boot.
>>>>>
>>>>> Waste of memory? You're only mapping an existing chunk of memory.
>>>>> DYM waste of address space?
>>>>
>>>> No, I really meant waste of memory. The current grant API in Linux is 
>>>> allocating one Linux Page per grant. Although the grant is always 4K, so 
>>>> we won't be able to make use of the 60K for anything as long as we use 
>>>> this page for a grant.
>>>>
>>>> So if the grant is pre-allocated (such as for PV block), we won't be 
>>>> able use nr_grant * 60KB memory.
>>>
>>> I still don't follow - grant mappings ought to be done into ballooned
>>> (i.e. empty) pages, i.e. no memory would get wasted unless there
>>> are too few balloon pages available.
>>
>> Everything balloon out is less memory that can be used by Linux. If we
>> are only using 1/15 of the balloon out page that a huge waste of memory
>> for me.
> 
> I still don't get it: A ballooned page does not represent any memory
> (and at least on x86 iirc we start PV domains with 8Mb sort-of-
> pre-ballooned space). Where is the loss you're talking about coming
> from? Or, wait - you don't have PoD (yet) on ARM, so you have no
> choice.

I'm not sure how PoD would affect the problem here...

An ARM guest can be considered as x86 HVM guest + PV drivers. At domain
creation, all the RAM is allocated for this domain.

Furthermore we have no memory hotplug support, so we would steal RAM
page every time we have to balloon out a page.

Regards,

-- 
Julien Grall

Re: [RFC 1/2] xen/mm: Clarify the granularity for each Frame Number

[Jan Beulich]

>>> On 12.08.15 at 14:57, <julien.grall@citrix.com> wrote:
> On 12/08/15 12:58, Jan Beulich wrote:
>>>>> On 12.08.15 at 13:13, <julien.grall@citrix.com> wrote:
>>> On 12/08/15 11:33, Jan Beulich wrote:
>>>>>>> On 12.08.15 at 11:57, <julien.grall@citrix.com> wrote:
>>>>> On 12/08/2015 08:16, Jan Beulich wrote:
>>>>>>>>> On 05.08.15 at 15:18, <julien.grall@citrix.com> wrote:
>>>>>>> When the backend is 64K, it will map the foreign 4K at the top of a 64K
>>>>>>> page. It's a waste of memory, but it's easier to implement and it's
>>>>>>> still and improvement compare to have Linux crashing at boot.
>>>>>>
>>>>>> Waste of memory? You're only mapping an existing chunk of memory.
>>>>>> DYM waste of address space?
>>>>>
>>>>> No, I really meant waste of memory. The current grant API in Linux is 
>>>>> allocating one Linux Page per grant. Although the grant is always 4K, so 
>>>>> we won't be able to make use of the 60K for anything as long as we use 
>>>>> this page for a grant.
>>>>>
>>>>> So if the grant is pre-allocated (such as for PV block), we won't be 
>>>>> able use nr_grant * 60KB memory.
>>>>
>>>> I still don't follow - grant mappings ought to be done into ballooned
>>>> (i.e. empty) pages, i.e. no memory would get wasted unless there
>>>> are too few balloon pages available.
>>>
>>> Everything balloon out is less memory that can be used by Linux. If we
>>> are only using 1/15 of the balloon out page that a huge waste of memory
>>> for me.
>> 
>> I still don't get it: A ballooned page does not represent any memory
>> (and at least on x86 iirc we start PV domains with 8Mb sort-of-
>> pre-ballooned space). Where is the loss you're talking about coming
>> from? Or, wait - you don't have PoD (yet) on ARM, so you have no
>> choice.
> 
> I'm not sure how PoD would affect the problem here...

You would have a way to start a guest with some unpopulated
space set aside, i.e. instead of ballooning out memory it could
use unpopulated pages to map the grants into.

Jan

[RFC 2/2] xen/public: grant-table: Specificy the size of the grant

[Julien Grall]

The grant is always 4KB irrespectively of the page granularity of the
guest.

Signed-off-by: Julien Grall <julien.grall@citrix.com>
---

Cc: Ian Campbell <ian.campbell@citrix.com>
Cc: Ian Jackson <ian.jackson@eu.citrix.com>
Cc: Jan Beulich <jbeulich@suse.com>
Cc: Keir Fraser <keir@xen.org>
Cc: Tim Deegan <tim@xen.org>
---
 xen/include/public/grant_table.h | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/xen/include/public/grant_table.h b/xen/include/public/grant_table.h
index e9393fd..67a250d 100644
--- a/xen/include/public/grant_table.h
+++ b/xen/include/public/grant_table.h
@@ -51,6 +51,11 @@
  * know the real machine address of a page it is sharing. This makes
  * it possible to share memory correctly with domains running in
  * fully virtualised memory.
+ *
+ * The size of a grant is always 4KB irrespectively of the page
+ * granularity of the guest. This is means that when the guest is using
+ * 64KB page granularity, it will have to split the page in 4KB chunks
+ * and request a grant for every of them.
  */
 
 /***********************************
-- 
2.1.4

Re: [RFC 2/2] xen/public: grant-table: Specificy the size of the grant

[Stefano Stabellini]

On Wed, 5 Aug 2015, Julien Grall wrote:
> The grant is always 4KB irrespectively of the page granularity of the
> guest.
> 
> Signed-off-by: Julien Grall <julien.grall@citrix.com>
> ---
> 
> Cc: Ian Campbell <ian.campbell@citrix.com>
> Cc: Ian Jackson <ian.jackson@eu.citrix.com>
> Cc: Jan Beulich <jbeulich@suse.com>
> Cc: Keir Fraser <keir@xen.org>
> Cc: Tim Deegan <tim@xen.org>
> ---
>  xen/include/public/grant_table.h | 5 +++++
>  1 file changed, 5 insertions(+)
> 
> diff --git a/xen/include/public/grant_table.h b/xen/include/public/grant_table.h
> index e9393fd..67a250d 100644
> --- a/xen/include/public/grant_table.h
> +++ b/xen/include/public/grant_table.h
> @@ -51,6 +51,11 @@
>   * know the real machine address of a page it is sharing. This makes
>   * it possible to share memory correctly with domains running in
>   * fully virtualised memory.
> + *
> + * The size of a grant is always 4KB irrespectively of the page
> + * granularity of the guest. This is means that when the guest is using
                                      ^ no is

> + * 64KB page granularity, it will have to split the page in 4KB chunks
> + * and request a grant for every of them.
                              ^ each

>   */
>  
>  /***********************************
> -- 
> 2.1.4
> 

Re: [RFC 2/2] xen/public: grant-table: Specificy the size of the grant

[Jan Beulich]

>>> On 05.08.15 at 13:28, <julien.grall@citrix.com> wrote:
> --- a/xen/include/public/grant_table.h
> +++ b/xen/include/public/grant_table.h
> @@ -51,6 +51,11 @@
>   * know the real machine address of a page it is sharing. This makes
>   * it possible to share memory correctly with domains running in
>   * fully virtualised memory.
> + *
> + * The size of a grant is always 4KB irrespectively of the page
> + * granularity of the guest. This is means that when the guest is using
> + * 64KB page granularity, it will have to split the page in 4KB chunks
> + * and request a grant for every of them.
>   */

This paragraph should get "on ARM" added somewhere. There is no
inherent requirement for such a restriction on another architecture
Xen might get ported to; the only restriction of the current interface
is that it can't support page sizes from 64k up. On an architecture
e.g. not supporting other than 8k pages things would still be fine,
yet the comment above would then be wrong.

An alternative might be to say "... is always the smallest
architecturally supported page size (e.g. 4k on x86 and ARM)
irrespective of ...".

Jan

Re: [RFC 2/2] xen/public: grant-table: Specificy the size of the grant

[Julien Grall]

On 12/08/2015 08:21, Jan Beulich wrote:
> This paragraph should get "on ARM" added somewhere. There is no
> inherent requirement for such a restriction on another architecture
> Xen might get ported to; the only restriction of the current interface
> is that it can't support page sizes from 64k up. On an architecture
> e.g. not supporting other than 8k pages things would still be fine,
> yet the comment above would then be wrong.
>
> An alternative might be to say "... is always the smallest
> architecturally supported page size (e.g. 4k on x86 and ARM)
> irrespective of ...".

While this is true today, I don't see anything that would prevent a new 
architecture to respect this.

What about replacing 4KB by "Xen Page size" and define this size per 
architecture in arch-*.h?

Regards,

-- 
Julien Grall

Re: [RFC 2/2] xen/public: grant-table: Specificy the size of the grant

[Jan Beulich]

>>> On 12.08.15 at 12:00, <julien.grall@citrix.com> wrote:

> 
> On 12/08/2015 08:21, Jan Beulich wrote:
>> This paragraph should get "on ARM" added somewhere. There is no
>> inherent requirement for such a restriction on another architecture
>> Xen might get ported to; the only restriction of the current interface
>> is that it can't support page sizes from 64k up. On an architecture
>> e.g. not supporting other than 8k pages things would still be fine,
>> yet the comment above would then be wrong.
>>
>> An alternative might be to say "... is always the smallest
>> architecturally supported page size (e.g. 4k on x86 and ARM)
>> irrespective of ...".
> 
> While this is true today, I don't see anything that would prevent a new 
> architecture to respect this.

I don't think I get what you're trying to say here.

> What about replacing 4KB by "Xen Page size" and define this size per 
> architecture in arch-*.h?

That looks okay to me.

Jan

Re: [RFC 2/2] xen/public: grant-table: Specificy the size of the grant

[Julien Grall]

On 12/08/15 11:35, Jan Beulich wrote:
>>>> On 12.08.15 at 12:00, <julien.grall@citrix.com> wrote:
> 
>>
>> On 12/08/2015 08:21, Jan Beulich wrote:
>>> This paragraph should get "on ARM" added somewhere. There is no
>>> inherent requirement for such a restriction on another architecture
>>> Xen might get ported to; the only restriction of the current interface
>>> is that it can't support page sizes from 64k up. On an architecture
>>> e.g. not supporting other than 8k pages things would still be fine,
>>> yet the comment above would then be wrong.
>>>
>>> An alternative might be to say "... is always the smallest
>>> architecturally supported page size (e.g. 4k on x86 and ARM)
>>> irrespective of ...".
>>
>> While this is true today, I don't see anything that would prevent a new 
>> architecture to respect this.
> 
> I don't think I get what you're trying to say here.

I meant that it may be possible to have an architecture where Xen would
support 64KB page size and the smallest page granularity would be 4KB.

Regards,


-- 
Julien Grall