32bit xen and "claim"

32bit xen and "claim"

[Dan Magenheimer]

Hi Keir and Jan --

With the plan to obsolete the x86 32-bit hypervisor at 4.3,
when prototyping the "claim" hypercall/subop, can I assume
that the CONFIG_X86 code in the hypervisor and, specifically
any separation of the concepts of xen_heap from dom_heap,
can be ignored?

Or will the ARM version of the hypervisor be requiring
a similar separation of xen_heap vs dom_heap?

If the CONFIG_X86/separation code will no longer be used,
shall I submit a patch to delete the code in page_alloc.c
and memory.c?  Or can you (Keir or Jan) just delete it?

Thanks,
Dan

Re: 32bit xen and "claim"

[Jan Beulich]

>>> Dan Magenheimer <dan.magenheimer@oracle.com> 11/01/12 9:34 PM >>>
>With the plan to obsolete the x86 32-bit hypervisor at 4.3,

That's not just a plan, but reality already.

>when prototyping the "claim" hypercall/subop, can I assume
>that the CONFIG_X86 code in the hypervisor and, specifically
>any separation of the concepts of xen_heap from dom_heap,
>can be ignored?

No, you shouldn't. Once adding support for memory amounts beyond 5Tb
I expect the separation to become meaningful even for x86-64.

>Or will the ARM version of the hypervisor be requiring
>a similar separation of xen_heap vs dom_heap?
>
>If the CONFIG_X86/separation code will no longer be used,
>shall I submit a patch to delete the code in page_alloc.c
>and memory.c?  Or can you (Keir or Jan) just delete it?

Please don't.

Jan

Re: 32bit xen and "claim"

[Dan Magenheimer]

> From: Jan Beulich [mailto:jbeulich@suse.com]
> Sent: Thursday, November 01, 2012 2:42 PM
> To: Dan Magenheimer; keir@xen.org
> Cc: xen-devel@lists.xen.org
> Subject: Re: 32bit xen and "claim"

> >If the CONFIG_X86/separation code will no longer be used,
> >shall I submit a patch to delete the code in page_alloc.c
> >and memory.c?  Or can you (Keir or Jan) just delete it?
> 
> Please don't.

OK.  I'm glad I asked rather than assuming.

> >when prototyping the "claim" hypercall/subop, can I assume
> >that the CONFIG_X86 code in the hypervisor and, specifically
> >any separation of the concepts of xen_heap from dom_heap,
> >can be ignored?
> 
> No, you shouldn't. Once adding support for memory amounts beyond 5Tb
> I expect the separation to become meaningful even for x86-64.

On quick scan, I don't see anything obvious in the archives
that explains why 5Tb is the limit (rather than, say, 4Tb
or 8Tb, or some other power of two).  Could you provide a
pointer to this info or, if you agree it is non-obvious and
undocumented, say a few words of explanation?

Also, just wondering, should exceeding 5Tb be on the 4.3
features list or is >5Tb physical memory still too far away?

Thanks,
Dan

Re: 32bit xen and "claim"

[Jan Beulich]

>>> On 01.11.12 at 21:55, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
>> >when prototyping the "claim" hypercall/subop, can I assume
>> >that the CONFIG_X86 code in the hypervisor and, specifically
>> >any separation of the concepts of xen_heap from dom_heap,
>> >can be ignored?
>> 
>> No, you shouldn't. Once adding support for memory amounts beyond 5Tb
>> I expect the separation to become meaningful even for x86-64.
> 
> On quick scan, I don't see anything obvious in the archives
> that explains why 5Tb is the limit (rather than, say, 4Tb
> or 8Tb, or some other power of two).  Could you provide a
> pointer to this info or, if you agree it is non-obvious and
> undocumented, say a few words of explanation?

xen/include/asm-x86/config.h:DIRECTMAP_* (and the comment
preceding all those #define-s).

> Also, just wondering, should exceeding 5Tb be on the 4.3
> features list or is >5Tb physical memory still too far away?

It _is_ on the feature list. So far I just had way too many other
issues to deal with, preventing me to get started with that work.

Jan

Re: 32bit xen and "claim"

[Dan Magenheimer]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Subject: RE: 32bit xen and "claim"
> 
> >>> On 01.11.12 at 21:55, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> >> >when prototyping the "claim" hypercall/subop, can I assume
> >> >that the CONFIG_X86 code in the hypervisor and, specifically
> >> >any separation of the concepts of xen_heap from dom_heap,
> >> >can be ignored?
> >>
> >> No, you shouldn't. Once adding support for memory amounts beyond 5Tb
> >> I expect the separation to become meaningful even for x86-64.
> >
> > On quick scan, I don't see anything obvious in the archives
> > that explains why 5Tb is the limit (rather than, say, 4Tb
> > or 8Tb, or some other power of two).  Could you provide a
> > pointer to this info or, if you agree it is non-obvious and
> > undocumented, say a few words of explanation?
> 
> xen/include/asm-x86/config.h:DIRECTMAP_* (and the comment
> preceding all those #define-s).

Sorry if this is a silly question, but the hex value
0xffff880000000000 looks like a Linux kernel base address
so it made me wonder:

Do I understand correctly that the 5TB limit only applies
because of legacy PV guests?
 
> > Also, just wondering, should exceeding 5Tb be on the 4.3
> > features list or is >5Tb physical memory still too far away?
> 
> It _is_ on the feature list. So far I just had way too many other
> issues to deal with, preventing me to get started with that work.

Sorry, I should have looked that up first. :-}

Does it make sense to have a runtime option that unsets the
physical limit but disallows legacy PV guests?  If this
defaults to false for machines with RAM<=5TB but to true
for machines with RAM>5TB, then the feature is "done"
(AND we have put a stake in the ground to begin the
slow obsolescence of PV functionality).

Just an idea...

Dan

Re: 32bit xen and "claim"

[Keir Fraser]

On 05/11/2012 19:16, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

>> xen/include/asm-x86/config.h:DIRECTMAP_* (and the comment
>> preceding all those #define-s).
> 
> Sorry if this is a silly question, but the hex value
> 0xffff880000000000 looks like a Linux kernel base address
> so it made me wonder:
> 
> Do I understand correctly that the 5TB limit only applies
> because of legacy PV guests?

Yes. For HVM guests we can direct-map all physical memory, all the time. We
will only ever need ephemeral mappings on x86/64 when running PV guests.

 -- Keir

Re: 32bit xen and "claim"

[Jan Beulich]

>>> On 05.11.12 at 20:16, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> Does it make sense to have a runtime option that unsets the
> physical limit but disallows legacy PV guests?  If this
> defaults to false for machines with RAM<=5TB but to true
> for machines with RAM>5TB, then the feature is "done"
> (AND we have put a stake in the ground to begin the
> slow obsolescence of PV functionality).

That would be interesting: Mukesh's PVH code wasn't even
posted yet, i.e. you're proposing to render systems with more
than 5Tb unbootable (for the lack of a - necessarily PV - Dom0
kernel runnable in that environment).

But yes, the plan is to extend the 1:1 mapping beyond 5Tb for
non-PV guests, and going through actual mapping code only
when acting in the context of a (64-bit) PV guest (even 32-bit
PV guests can have the full 1:1 mapping).

Jan

Re: 32bit xen and "claim"

[Dan Magenheimer]

> From: Jan Beulich [mailto:JBeulich@suse.com]
> Sent: Tuesday, November 06, 2012 2:44 AM
> To: Dan Magenheimer
> Cc: xen-devel@lists.xen.org; keir@xen.org
> Subject: RE: 32bit xen and "claim"
> 
> >>> On 05.11.12 at 20:16, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
> > Does it make sense to have a runtime option that unsets the
> > physical limit but disallows legacy PV guests?  If this
> > defaults to false for machines with RAM<=5TB but to true
> > for machines with RAM>5TB, then the feature is "done"
> > (AND we have put a stake in the ground to begin the
> > slow obsolescence of PV functionality).
> 
> That would be interesting: Mukesh's PVH code wasn't even
> posted yet, i.e. you're proposing to render systems with more
> than 5Tb unbootable (for the lack of a - necessarily PV - Dom0
> kernel runnable in that environment).

Good point.  BUT... couldn't a PV dom0 started with dom0_mem=X
(where X is smaller than 5GB) still work?
 
> But yes, the plan is to extend the 1:1 mapping beyond 5Tb for
> non-PV guests, and going through actual mapping code only
> when acting in the context of a (64-bit) PV guest (even 32-bit
> PV guests can have the full 1:1 mapping).

That makes sense.  I guess I am just worried that this will
require enough surgery that there will be a long bug tail.
And, since 5TB machines will be quite rare for the next
year or so, deprecating PV domains on those machines earlier
than on smaller machines might have a much smaller impact
on the Xen community.

But this is all just a suggestion... I will stop now and
leave it in your capable hands.

Dan

Re: 32bit xen and "claim"

[Jan Beulich]

>>> On 06.11.12 at 21:40, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
>>  From: Jan Beulich [mailto:JBeulich@suse.com]
>> Sent: Tuesday, November 06, 2012 2:44 AM
>> To: Dan Magenheimer
>> Cc: xen-devel@lists.xen.org; keir@xen.org 
>> Subject: RE: 32bit xen and "claim"
>> 
>> >>> On 05.11.12 at 20:16, Dan Magenheimer <dan.magenheimer@oracle.com> wrote:
>> > Does it make sense to have a runtime option that unsets the
>> > physical limit but disallows legacy PV guests?  If this
>> > defaults to false for machines with RAM<=5TB but to true
>> > for machines with RAM>5TB, then the feature is "done"
>> > (AND we have put a stake in the ground to begin the
>> > slow obsolescence of PV functionality).
>> 
>> That would be interesting: Mukesh's PVH code wasn't even
>> posted yet, i.e. you're proposing to render systems with more
>> than 5Tb unbootable (for the lack of a - necessarily PV - Dom0
>> kernel runnable in that environment).
> 
> Good point.  BUT... couldn't a PV dom0 started with dom0_mem=X
> (where X is smaller than 5GB) still work?

No - Xen would still need to be able to access all memory when in
the context of such a Dom0 (even address restricting that Dom0
just like we do for 32-bit PV guests wouldn't help, as e.g. granted
pages from DomU-s may need accessing in Xen in the context of
Dom0).

Jan

Re: 32bit xen and "claim"

[Tim Deegan]

At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote:
> With the plan to obsolete the x86 32-bit hypervisor at 4.3,
> when prototyping the "claim" hypercall/subop, can I assume
> that the CONFIG_X86 code in the hypervisor and, specifically
> any separation of the concepts of xen_heap from dom_heap,
> can be ignored?
> 
> Or will the ARM version of the hypervisor be requiring
> a similar separation of xen_heap vs dom_heap?

Yes, 32-bit ARM has this separation.

Tim.

Re: 32bit xen and "claim"

[Dan Magenheimer]

> From: Tim Deegan [mailto:tim@xen.org]
> Subject: Re: [Xen-devel] 32bit xen and "claim"
> 
> At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote:
> > With the plan to obsolete the x86 32-bit hypervisor at 4.3,
> > when prototyping the "claim" hypercall/subop, can I assume
> > that the CONFIG_X86 code in the hypervisor and, specifically
> > any separation of the concepts of xen_heap from dom_heap,
> > can be ignored?
> >
> > Or will the ARM version of the hypervisor be requiring
> > a similar separation of xen_heap vs dom_heap?
> 
> Yes, 32-bit ARM has this separation.

Hmmm... looking at page_alloc.c... does ARM overload CONFIG_X86
to mean CONFIG-32-bitness then?  Seems dangerous given that
various random chunks of CONFIG_X86 may be disappearing.

Re: 32bit xen and "claim"

[Tim Deegan]

At 13:57 -0700 on 01 Nov (1351778261), Dan Magenheimer wrote:
> > From: Tim Deegan [mailto:tim@xen.org]
> > Subject: Re: [Xen-devel] 32bit xen and "claim"
> > 
> > At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote:
> > > With the plan to obsolete the x86 32-bit hypervisor at 4.3,
> > > when prototyping the "claim" hypercall/subop, can I assume
> > > that the CONFIG_X86 code in the hypervisor and, specifically
> > > any separation of the concepts of xen_heap from dom_heap,
> > > can be ignored?
> > >
> > > Or will the ARM version of the hypervisor be requiring
> > > a similar separation of xen_heap vs dom_heap?
> > 
> > Yes, 32-bit ARM has this separation.
> 
> Hmmm... looking at page_alloc.c... does ARM overload CONFIG_X86
> to mean CONFIG-32-bitness then? 

No.  CONFIG_X86 doesn't mean 32-bit, even on x86; it means i386/amd64 as
distinct from MIPS/ARM/PPC.

Tim.

Re: 32bit xen and "claim"

[Dan Magenheimer]

> From: Tim Deegan [mailto:tim@xen.org]
> Sent: Friday, November 02, 2012 3:31 AM
> To: Dan Magenheimer
> Cc: Keir (Xen.org); Jan Beulich; xen-devel@lists.xen.org
> Subject: Re: [Xen-devel] 32bit xen and "claim"
> 
> At 13:57 -0700 on 01 Nov (1351778261), Dan Magenheimer wrote:
> > > From: Tim Deegan [mailto:tim@xen.org]
> > > Subject: Re: [Xen-devel] 32bit xen and "claim"
> > >
> > > At 13:34 -0700 on 01 Nov (1351776880), Dan Magenheimer wrote:
> > > > With the plan to obsolete the x86 32-bit hypervisor at 4.3,
> > > > when prototyping the "claim" hypercall/subop, can I assume
> > > > that the CONFIG_X86 code in the hypervisor and, specifically
> > > > any separation of the concepts of xen_heap from dom_heap,
> > > > can be ignored?
> > > >
> > > > Or will the ARM version of the hypervisor be requiring
> > > > a similar separation of xen_heap vs dom_heap?
> > >
> > > Yes, 32-bit ARM has this separation.
> >
> > Hmmm... looking at page_alloc.c... does ARM overload CONFIG_X86
> > to mean CONFIG-32-bitness then?
> 
> No.  CONFIG_X86 doesn't mean 32-bit, even on x86; it means i386/amd64 as
> distinct from MIPS/ARM/PPC.

Oops, sorry, I was confusing the old code in page_alloc.c that
said:

	#if !defined(__x86_64__) && !defined(__ia64__)

when I was reading the new code that says:

	#if !defined(CONFIG_X86)

(why not #ifndef?)

Sorry for the ARM-y noise. :-(

But as long as I've bothered you already... does the ARM port
already (or will it soon) support 64-bit versions of ARM?

Dan

Re: 32bit xen and "claim"

[Tim Deegan]

At 11:00 -0800 on 05 Nov (1352113204), Dan Magenheimer wrote:
> But as long as I've bothered you already... does the ARM port
> already (or will it soon) support 64-bit versions of ARM?

Real Soon Now, we hope.  We've been a bit distracted by having actual
hardware to test the 32-bit version on, but 64-bit support is definitely
planned.

Tim.

Re: 32bit xen and "claim"

[Keir Fraser]

On 05/11/2012 19:00, "Dan Magenheimer" <dan.magenheimer@oracle.com> wrote:

> #if !defined(CONFIG_X86)
> 
> (why not #ifndef?)

We have no coding convention to prefer one over the other. Personally I
would say they are equally readable.

 -- Keir