dom0 boot failure with 256G

dom0 boot failure with 256G

[Mukesh Rathor]

Hi,

    I've been intensely debugging a dom0 crash right at boot in
    pagetable_init() with 256G. We are running 64bit hyp with 32bit dom0.
    This on xen 3.1.3, which seems to have fixes from changeset 16548.

    It boils down to this:

   crash occurs in dom0: mfn_to_pfn() during machine_to_phys_mapping[mfn]
   lookup  where mfn == 0x3f2ec09 is too big. The table starts at: 0xf5800000
   which I see comes from MACH2PHYS_VIRT_START.

   Trying to figure the hyp out, I notice phys addr is set as:

   d->arch.physaddr_bitsize =
           fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
             + (PAGE_SIZE - 2);
   which is set to 0x1019, totally baffling me. I'd expect it to be 32,
   36, or 64????????

  moving along, in alloc_domheap_pages() :
    bits = domain_clamp_alloc_bitsize(d, bits ? : (BITS_PER_LONG+PAGE_SHIFT));

   domain_clamp_alloc_bitsize(struct domain *d, unsigned int bits)
   {
       if ( (d == NULL) || !is_pv_32on64_domain(d) )
               return bits;
       return min(d->arch.physaddr_bitsize, bits);
            ---> here : physaddr: 0x1019, bits:76 ?????
            ---> i'm thinking bits should be 36+12 == 48 ??
   }

    finally, call to alloc_heap_pages(...) returns frames too high, pg
    ptr is returned at : ffff82849df30000 (zone_hi == 0x27)


    On a side note, is my understanding correct that while a guest pfn must
    always be < 36 bits for a PAE 32 guest, the mfn doesn't have to be?


    thanks
    mukesh

RE: dom0 boot failure with 256G

[Ian Pratt]

>     I've been intensely debugging a dom0 crash right at boot in
>     pagetable_init() with 256G. We are running 64bit hyp with 32bit
> dom0.
>     This on xen 3.1.3, which seems to have fixes from changeset 16548.

256GB is more than a 32b dom0 can handle. Presumably it works if you set
dom0_mem= to something smaller?

We should probably make the clipping of dom0 memory to something like
64GB automatic.

Ian

 
>     It boils down to this:
> 
>    crash occurs in dom0: mfn_to_pfn() during
> machine_to_phys_mapping[mfn]
>    lookup  where mfn == 0x3f2ec09 is too big. The table starts at:
> 0xf5800000
>    which I see comes from MACH2PHYS_VIRT_START.
> 
>    Trying to figure the hyp out, I notice phys addr is set as:
> 
>    d->arch.physaddr_bitsize =
>            fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
>              + (PAGE_SIZE - 2);
>    which is set to 0x1019, totally baffling me. I'd expect it to be
32,
>    36, or 64????????
> 
>   moving along, in alloc_domheap_pages() :
>     bits = domain_clamp_alloc_bitsize(d, bits ? :
> (BITS_PER_LONG+PAGE_SHIFT));
> 
>    domain_clamp_alloc_bitsize(struct domain *d, unsigned int bits)
>    {
>        if ( (d == NULL) || !is_pv_32on64_domain(d) )
>                return bits;
>        return min(d->arch.physaddr_bitsize, bits);
>             ---> here : physaddr: 0x1019, bits:76 ?????
>             ---> i'm thinking bits should be 36+12 == 48 ??
>    }
> 
>     finally, call to alloc_heap_pages(...) returns frames too high, pg
>     ptr is returned at : ffff82849df30000 (zone_hi == 0x27)
> 
> 
>     On a side note, is my understanding correct that while a guest pfn
> must
>     always be < 36 bits for a PAE 32 guest, the mfn doesn't have to
be?
> 
> 
>     thanks
>     mukesh
> 
> 
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xensource.com
> http://lists.xensource.com/xen-devel

Re: dom0 boot failure with 256G

[Keir Fraser]

On 5/6/08 05:00, "Ian Pratt" <Ian.Pratt@eu.citrix.com> wrote:

>>     I've been intensely debugging a dom0 crash right at boot in
>>     pagetable_init() with 256G. We are running 64bit hyp with 32bit
>> dom0.
>>     This on xen 3.1.3, which seems to have fixes from changeset 16548.
> 
> 256GB is more than a 32b dom0 can handle. Presumably it works if you set
> dom0_mem= to something smaller?
> 
> We should probably make the clipping of dom0 memory to something like
> 64GB automatic.

Xen ought to have crashed during dom0 build in that case, being unable to
allocate enough memory of the restricted address width supported by the 32b
dom0 guest. I agree that dom0_mem should be automatically intelligently
clamped. Currently nr_pages is clamped to UINT_MAX (!) for a 32b dom0, which
is still rather a big number.

I'll have to sketch out exactly what that physaddr_bitsize computation is
doing and see if I can make sense of it...

 -- Keir

Re: dom0 boot failure with 256G

[Keir Fraser]

On 5/6/08 03:07, "Mukesh Rathor" <mukesh.rathor@oracle.com> wrote:

>    d->arch.physaddr_bitsize =
>            fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
>              + (PAGE_SIZE - 2);
>    which is set to 0x1019, totally baffling me. I'd expect it to be 32,
>    36, or 64????????

PAGE_SIZE should be PAGE_SHIFT. If you fix that then you may discover that
dom0 can no longer be built (since there is not enough memory below 64GB to
build a ~256GB dom0, which is the default that the dom0 builder will go for)
so make sure you have a suitably restrictive dom0_mem= option on your Xen
command line. As Ian says we should fix that more nicely.

 -- Keir

[PATCH] Re: dom0 boot failure with 256G

[Jan Beulich]

>   Trying to figure the hyp out, I notice phys addr is set as:
>
>   d->arch.physaddr_bitsize =
>           fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
>             + (PAGE_SIZE - 2);
>   which is set to 0x1019, totally baffling me. I'd expect it to be 32,
>   36, or 64????????

Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up
- sorry, my fault.

Signed-off-by: Jan Beulich <jbeulich@novell.com>

Index: 2008-05-08/xen/arch/x86/domain.c
===================================================================
--- 2008-05-08.orig/xen/arch/x86/domain.c	2008-05-09 12:19:01.000000000 +0200
+++ 2008-05-08/xen/arch/x86/domain.c	2008-06-05 09:31:30.000000000 +0200
@@ -353,7 +353,7 @@ int switch_compat(struct domain *d)
 
     d->arch.physaddr_bitsize =
         fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
-        + (PAGE_SIZE - 2);
+        + (PAGE_SHIFT - 2);
 
     return 0;
 
Index: 2008-05-08/xen/arch/x86/domain_build.c
===================================================================
--- 2008-05-08.orig/xen/arch/x86/domain_build.c	2008-06-04 08:26:22.000000000 +0200
+++ 2008-05-08/xen/arch/x86/domain_build.c	2008-06-05 09:31:36.000000000 +0200
@@ -366,7 +366,7 @@ int __init construct_dom0(
     if ( is_pv_32on64_domain(d) )
         d->arch.physaddr_bitsize =
             fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
-            + (PAGE_SIZE - 2);
+            + (PAGE_SHIFT - 2);
 #endif
 
     /*

Re: dom0 boot failure with 256G

[Keir Fraser]

On 5/6/08 08:24, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:

>> 256GB is more than a 32b dom0 can handle. Presumably it works if you set
>> dom0_mem= to something smaller?
>> 
>> We should probably make the clipping of dom0 memory to something like
>> 64GB automatic.
> 
> Xen ought to have crashed during dom0 build in that case, being unable to
> allocate enough memory of the restricted address width supported by the 32b
> dom0 guest. I agree that dom0_mem should be automatically intelligently
> clamped. Currently nr_pages is clamped to UINT_MAX (!) for a 32b dom0, which
> is still rather a big number.

Actually compat guests are good for up to 128GB, so you should be fine with
no dom0_mem= parameter.

 -- Keir

Re: dom0 boot failure with 256G

[Jeremy Fitzhardinge]

Keir Fraser wrote:
> Actually compat guests are good for up to 128GB, so you should be fine with
> no dom0_mem= parameter.
>   

Real PAE processors are limited to 2^36 = 64GB, and the guest kernel may 
mask ptes at that point.

    J

Re: dom0 boot failure with 256G

[Keir Fraser]

On 5/6/08 13:54, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:

> Keir Fraser wrote:
>> Actually compat guests are good for up to 128GB, so you should be fine with
>> no dom0_mem= parameter.
>>   
> 
> Real PAE processors are limited to 2^36 = 64GB, and the guest kernel may
> mask ptes at that point.

A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug
we've bumped into once or twice.

 -- Keir

Re: dom0 boot failure with 256G

[Jan Beulich]

>>> Keir Fraser <keir.fraser@eu.citrix.com> 05.06.08 14:58 >>>
>On 5/6/08 13:54, "Jeremy Fitzhardinge" <jeremy@goop.org> wrote:
>
>> Keir Fraser wrote:
>>> Actually compat guests are good for up to 128GB, so you should be fine with
>>> no dom0_mem= parameter.
>>>   
>> 
>> Real PAE processors are limited to 2^36 = 64GB, and the guest kernel may
>> mask ptes at that point.
>
>A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug
>we've bumped into once or twice.

Even a native kernel ought to honor the physical address width the
processor supports rather than arbitrarily masking off bits just because
original PAE only supported 36 bits.

Jan

Re: dom0 boot failure with 256G

[Keir Fraser]

On 5/6/08 14:15, "Jan Beulich" <jbeulich@novell.com> wrote:

>> A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug
>> we've bumped into once or twice.
> 
> Even a native kernel ought to honor the physical address width the
> processor supports rather than arbitrarily masking off bits just because
> original PAE only supported 36 bits.

Yes, both Intel and AMD have wider PAE addressing these days.

 -- Keir

Re: dom0 boot failure with 256G

[Jeremy Fitzhardinge]

Jan Beulich wrote:
>> A paravirtualised kernel needs to have its PAGE_MASK widened. This is a bug
>> we've bumped into once or twice.
>>     
>
> Even a native kernel ought to honor the physical address width the
> processor supports rather than arbitrarily masking off bits just because
> original PAE only supported 36 bits.

Current 2.6 is too strict about this.  I'll queue up a patch to fix it.

    J

Re: dom0 boot failure with 256G

[mukesh rathor]

Ian Pratt <Ian.Pratt <at> eu.citrix.com> writes:

> 
> >     I've been intensely debugging a dom0 crash right at boot in
> >     pagetable_init() with 256G. We are running 64bit hyp with 32bit
> > dom0.
> >     This on xen 3.1.3, which seems to have fixes from changeset 16548.
> 
> 256GB is more than a 32b dom0 can handle. Presumably it works if you set
> dom0_mem= to something smaller?
> 
> We should probably make the clipping of dom0 memory to something like
> 64GB automatic.
> 
> Ian
> 
>  
..........

My apologies, I forgot to mention dom0_mem is set to : dom0_mem=569M

(XEN) Command line: console=com2,ttyS0 com2=115200,8n1 earlykdb dom0_mem=569M
mem=256G

Things are fine with mem=128G. I've not tried 192G or other number. I just
started debugging.....

I'll try with :
>    d->arch.physaddr_bitsize =
> >            fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
> >              + (PAGE_SIZE - 2);
> >    which is set to 0x1019, totally baffling me. I'd expect it to be 32,
> >    36, or 64????????

>PAGE_SIZE should be PAGE_SHIFT. If you fix that then you may discover ...

thanks
mukesh

Re: [PATCH] Re: dom0 boot failure with 256G

[Mukesh Rathor]

Jan Beulich wrote:
>>   Trying to figure the hyp out, I notice phys addr is set as:
>>
>>   d->arch.physaddr_bitsize =
>>           fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
>>             + (PAGE_SIZE - 2);
>>   which is set to 0x1019, totally baffling me. I'd expect it to be 32,
>>   36, or 64????????
> 
> Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up
> - sorry, my fault.
> 

Yup, that fixed it. Can you pl explain in couple sentences how we are getting 
at the physaddr_bitsize here?
I still don't understand why 37 is an OK size for a 32-PAE guest (i thought 36 
was max).

Thanks,
Mukesh

Re: dom0 boot failure with 256G

[Mukesh Rathor]

Keir Fraser wrote:
> 
> 
> On 5/6/08 08:24, "Keir Fraser" <keir.fraser@eu.citrix.com> wrote:
> 
>>> 256GB is more than a 32b dom0 can handle. Presumably it works if you set
>>> dom0_mem= to something smaller?
>>>
>>> We should probably make the clipping of dom0 memory to something like
>>> 64GB automatic.
>> Xen ought to have crashed during dom0 build in that case, being unable to
>> allocate enough memory of the restricted address width supported by the 32b
>> dom0 guest. I agree that dom0_mem should be automatically intelligently
>> clamped. Currently nr_pages is clamped to UINT_MAX (!) for a 32b dom0, which
>> is still rather a big number.
> 
> Actually compat guests are good for up to 128GB, so you should be fine with
> no dom0_mem= parameter.
> 
>  -- Keir
> 

No, it crashed without dom0_mem paramater after I fixed the physaddr_bitsize 
in construct_dom0(). It looked like it was allocating almost 256G to dom0... 
things are fine now with dom0_mem= specified.

thanks
mukesh

Re: [PATCH] Re: dom0 boot failure with 256G

[Jan Beulich]

>>> Mukesh Rathor <mukesh.rathor@oracle.com> 06.06.08 04:43 >>>
>
>Jan Beulich wrote:
>>>   Trying to figure the hyp out, I notice phys addr is set as:
>>>
>>>   d->arch.physaddr_bitsize =
>>>           fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
>>>             + (PAGE_SIZE - 2);
>>>   which is set to 0x1019, totally baffling me. I'd expect it to be 32,
>>>   36, or 64????????
>> 
>> Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up
>> - sorry, my fault.
>> 
>
>Yup, that fixed it. Can you pl explain in couple sentences how we are getting 
>at the physaddr_bitsize here?

The whole hypervisor hole is used solely for the (compatibility) m2p map
when running on a 64-bit hypervisor. Hence the size of that hole, rounded
down to a power of two, determines the address restriction a domain
must be assigned. A kernel could theoretically specify being able to
tolerate a larger hole (though XEN_ELFNOTE_HV_START_LOW), so to
increase the physical address width it can get memory from.

>I still don't understand why 37 is an OK size for a 32-PAE guest (i thought 36 
>was max).

36 was the original max, but even on native hardware you can go beyond
that nowadays. On 64-bit Xen, this was a natural extension even
independent of confirming hardware support of wider than 36-bit physical
addresses for 32-bit code.

Of course, we all realize that giving dom0 128G won't do any good, afair
even native Linux can't reasonably deal with 64G (due to the huge
memmap needed in lowmem). But allowing an as wide as possible range
to get memory from for the domain is certainly very desirable (I am
actually considering ideas on how to overcome even that limit, as in
practical uses this is really undesirable to have, since there's no way
to balloon out specific memory [i.e. such below the 37-bit address
boundary] from other domains).

Jan

Re: [PATCH] Re: dom0 boot failure with 256G

[Mukesh Rathor]

Jan Beulich wrote:
>>>> Mukesh Rathor <mukesh.rathor@oracle.com> 06.06.08 04:43 >>>
>> Jan Beulich wrote:
>>>>   Trying to figure the hyp out, I notice phys addr is set as:
>>>>
>>>>   d->arch.physaddr_bitsize =
>>>>           fls((1UL << 32) - HYPERVISOR_COMPAT_VIRT_START(d)) - 1
>>>>             + (PAGE_SIZE - 2);
>>>>   which is set to 0x1019, totally baffling me. I'd expect it to be 32,
>>>>   36, or 64????????
>>> Of course it should be PAGE_SHIFT in here! c/s 14097 screwed this up
>>> - sorry, my fault.
>>>
>> Yup, that fixed it. Can you pl explain in couple sentences how we are getting 
>> at the physaddr_bitsize here?
> 
> The whole hypervisor hole is used solely for the (compatibility) m2p map
> when running on a 64-bit hypervisor. Hence the size of that hole, rounded
> down to a power of two, determines the address restriction a domain
> must be assigned. A kernel could theoretically specify being able to
> tolerate a larger hole (though XEN_ELFNOTE_HV_START_LOW), so to
> increase the physical address width it can get memory from.
> 
>> I still don't understand why 37 is an OK size for a 32-PAE guest (i thought 36 
>> was max).
> 
> 36 was the original max, but even on native hardware you can go beyond
> that nowadays. On 64-bit Xen, this was a natural extension even
> independent of confirming hardware support of wider than 36-bit physical
> addresses for 32-bit code.
> 
> Of course, we all realize that giving dom0 128G won't do any good, afair
> even native Linux can't reasonably deal with 64G (due to the huge
> memmap needed in lowmem). But allowing an as wide as possible range
> to get memory from for the domain is certainly very desirable (I am
> actually considering ideas on how to overcome even that limit, as in
> practical uses this is really undesirable to have, since there's no way
> to balloon out specific memory [i.e. such below the 37-bit address
> boundary] from other domains).
> 
> Jan

Got it. Thanks a lot, appreciate your time.

Mukesh